Why makes calling error or done in a BodyParser's Iteratee the request hang in Play Framework 2.0? - scala

I am trying to understand the reactive I/O concepts of Play 2.0 framework. In order to get a better understanding from the start I decided to skip the framework's helpers to construct iteratees of different kinds and to write a custom Iteratee from scratch to be used by a BodyParser to parse a request body.
Starting with the information available in Iteratees and ScalaBodyParser docs and two presentations about play reactive I/O this is what I came up with:
import play.api.mvc._
import play.api.mvc.Results._
import play.api.libs.iteratee.{Iteratee, Input}
import play.api.libs.concurrent.Promise
import play.api.libs.iteratee.Input.{El, EOF, Empty}
01 object Upload extends Controller {
02 def send = Action(BodyParser(rh => new SomeIteratee)) { request =>
03 Ok("Done")
04 }
05 }
06
07 case class SomeIteratee(state: Symbol = 'Cont, input: Input[Array[Byte]] = Empty, received: Int = 0) extends Iteratee[Array[Byte], Either[Result, Int]] {
08 println(state + " " + input + " " + received)
09
10 def fold[B](
11 done: (Either[Result, Int], Input[Array[Byte]]) => Promise[B],
12 cont: (Input[Array[Byte]] => Iteratee[Array[Byte], Either[Result, Int]]) => Promise[B],
13 error: (String, Input[Array[Byte]]) => Promise[B]
14 ): Promise[B] = state match {
15 case 'Done => { println("Done"); done(Right(received), Input.Empty) }
16 case 'Cont => cont(in => in match {
17 case in: El[Array[Byte]] => copy(input = in, received = received + in.e.length)
18 case Empty => copy(input = in)
19 case EOF => copy(state = 'Done, input = in)
20 case _ => copy(state = 'Error, input = in)
21 })
22 case _ => { println("Error"); error("Some error.", input) }
23 }
24 }
(Remark: All these things are new to me, so please forgive if something about this is total crap.)
The Iteratee is pretty dumb, it just reads all chunks, sums up the number of received bytes and prints out some messages. Everything works as expected when I call the controller action with some data - I can observe all chunks are received by the Iteratee and when all data is read it switches to state done and the request ends.
Now I started to play around with the code because I wanted to see the behaviour for these two cases:
Switching into state error before all input is read.
Switching into state done before all input is read and returning a Result instead of the Int.
My understanding of the documentation mentioned above is that both should be possible but actually I am not able to understand the observed behaviour. To test the first case I changed line 17 of the above code to:
17 case in: El[Array[Byte]] => copy(state = if(received + in.e.length > 10000) 'Error else 'Cont, input = in, received = received + in.e.length)
So I just added a condition to switch into the error state if more than 10000 bytes were received. The output I get is this:
'Cont Empty 0
'Cont El([B#38ecece6) 8192
'Error El([B#4ab50d3c) 16384
Error
Error
Error
Error
Error
Error
Error
Error
Error
Error
Error
Then the request hangs forever and never ends. My expectation from the above mentioned docs was that when I call the error function inside fold of an Iteratee the processing should be stopped. What is happening here is that the Iteratee's fold method is called several times after error has been called - well and then the request hangs.
When I switch into done state before reading all input the behaviour is quite similar. Changing line 15 to:
15 case 'Done => { println("Done with " + input); done(if (input == EOF) Right(received) else Left(BadRequest), Input.Empty) }
and line 17 to:
17 case in: El[Array[Byte]] => copy(state = if(received + in.e.length > 10000) 'Done else 'Cont, input = in, received = received + in.e.length)
produces the following output:
'Cont Empty 0
'Cont El([B#16ce00a8) 8192
'Done El([B#2e8d214a) 16384
Done with El([B#2e8d214a)
Done with El([B#2e8d214a)
Done with El([B#2e8d214a)
Done with El([B#2e8d214a)
and again the request hangs forever.
My main question is why the request is hanging in the above mentioned cases. If anybody could shed light on this I would greatly appreciate it!

Your understanding is perfectly right and I have just push a fix to master:
https://github.com/playframework/Play20/commit/ef70e641d9114ff8225332bf18b4dd995bd39bcc
Fixed both cases plus exceptions in the Iteratees.
Nice use of copy in case class for doing an Iteratee BTW.

Things must have changed with Play 2.1 - Promise is no longer parametric, and this example no longer compiles.

Related

Response entity was not subscribed after 100 seconds. Make sure to read the response entity body or call `discardBytes()` on it

I am building a scala application. Within the application, we are making a call to an external service, and fetching the data.
When I am hitting the endpoint of this external service using the
postman, I am getting the complete data, around 9000 lines of JSON
data, within 9 seconds.
But when I am hitting the same endpoint through my scala application, I am getting a 200 OK response, but getting the below error:
[WARN] [06/09/2022 18:05:45.765] [default-akka.actor.default-dispatcher-9] [default/Pool(shared->http://ad-manager-api-production.ap-south-1.elasticbeanstalk.com:80)] [4 (WaitingForResponseEntitySubscription)] Response entity was not subscribed after 100 seconds. Make sure to read the response entity body or call `discardBytes()` on it. GET /admin/campaigns Empty -> 200 OK Chunked
I read about it and found that we can set response-entity-subscription-timeout property to a higher value. I set it to about 100 seconds, but this does not seem to help.
My Code:
private val sendAndReceive = customSendAndReceive.getOrElse(HttpClientUtils.singleRequest)
.
.
.
def getActiveCampaigns: GetActiveCampaigns = () => {
val request = HttpRequest(
uri = s"$endpoint/admin/campaigns?status=PUBLISHED", // includes both PUBLISHED_READY and PUBLISHED_PAUSED
method = HttpMethods.GET,
headers = heathers
)
sendAndReceive(request).timed(getActiveCampaignsTimer).flatMap {
case HttpResponse(StatusCodes.OK, _, entity, _) =>
Unmarshal(entity).to[List[CampaignListDetailsDto]]
case response#HttpResponse(_, _, _, _) =>
response.discardEntityBytes()
Future.failed(new RuntimeException(s"Ad manager service exception: $response"))
case response =>
log.error(s"Error calling ad manager service: $response")
response.discardEntityBytes()
Future.failed(new RuntimeException(s"Ad manager service exception: $response"))
}
}
.
.
.
def getCampaignSpendData(getActiveCampaigns: GetActiveCampaigns, getCampaignTotalSpend: GetCampaignTotalSpend)(implicit ec: ExecutionContext): GetCampaignsSpendData = () => {
getActiveCampaigns()
.andThen {
case Failure(t) => log.error("Failed to fetch ads from ad manager", t)
}
.flatMap {
campaignList => Future.sequence(campaignList.map(campaign => budgetSpendPercentage(getCampaignTotalSpend)(campaign)))
}
}
Questions
What does this error exactly mean? Is it that it is able to connect to the endpoint but not able to get the complete data from it before the connection is closed/reset?
How can we address this issue?

Akka-Http client: How to get binary data from an http response?

I call an API to get a zip file response. The API responds correctly but I am unable to get the byte array from response because the future that should complete on getting the ByteString never completes:
val authorization = akka.http.javadsl.model.headers.Authorization.basic("xxxxx", "xxxxxx")
val query = Map("fed" -> "xxxx", "trd" -> "yyy", "id" -> "zzz")
val request = HttpRequest(HttpMethods.GET, Uri("https://xxxx.yyyy.com/ggg/ttt.php").withQuery(Query(params = query))).addHeader(authorization)
val responseFut = http.singleRequest(request)
responseFut1.map(response => {
println("*******************************")
println(response)
response.status match {
case akka.http.javadsl.model.StatusCodes.OK => {
println("^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^" + response._3)
val entityFut = response.entity.toStrict(60.seconds)
val byteStringFut = entityFut.flatMap(entity => {
entity.dataBytes.runFold(ByteString.empty)(_ ++ _)
})
println("#############################")
try {
byteStringFut.map(x => {
//this never prints =======================================problem
println("----------------------------" + x.toArray[Byte])
})
}catch{
case e: Exception => println("Error: " + e)
}
}
case _ => {}
}
})
If I print out the response this is what it looks like:
*******************************
HttpResponse(200 OK,List(Date: Fri, 08 Sep 2017 20:58:43 GMT, Server: Apache/2.4.18 (Ubuntu), Content-Disposition: attachment; filename="xxxxx.zip", Pragma: public, Cache-Contr
ol: public, must-revalidate, Content-Transfer-Encoding: binary),HttpEntity.Chunked(application/x-zip),HttpProtocol(HTTP/1.1))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^HttpEntity.Chunked(application/x-zip)
#############################
So response is coming back fine, but I still cannot get the binary data for the zip file.
We use akka-http in other places to call APIs that return json response and this approach seems to work fine there.
Why doesnt it work here? What am I doing wrong?
Any advice is appreciated.
Thanks.
Update
Adding byteStringFut.failed.foreach(println(_)) shows this exception: akka.http.scaladsl.model.EntityStreamException: HTTP chunk size exceeds the configured limit of 1048576 bytes
It looks like something went wrong and an exception was thrown in the async computation. You can check the exception by inspecting Future in the following way:
byteStringFut.failed.foreach(println)

How to check for 200 OK response status using Scala and Play Framework

I have a following Actor class that is responsible for sending a JSON message to a URL using POST.
import play.api.libs.ws._
class Worker extends Actor {
val logger: Logger = Logger("superman")
val supermanURL = "http://localhost:9000/superman/send"
def receive = {
case message: JsValue => {
val transactionID = (message \ "transactionID").get
println("Received JSON Object =>" + message)
val responseFromSuperman = WS.url(supermanURL).withHeaders("Content-Type" -> "application/json").post(message)
responseFromSuperman.map(
result => {
//TODO: Make sure to only log if response status is 200 OK
logger.info("""Message="ACK received from Superman" for transactionID=""" + transactionID)}
).recover { case error: Throwable =>
logger.error("""Message="NACK received from Superman" for transactionID=""" + transactionID) + " errorMessage:" + error.getLocalizedMessage()
}
}
}
}
So, if you look into my TODO above, I would like to add a check for a response type 200 OK. The current implementation is not doing that and it logs the message even if I manually send in a BadRequest. I tried checking for result.allHeaders which returns:
Map(Date -> Buffer(Wed, 27 Jan 2016 21:45:31 GMT), Content-Type -> Buffer(text/plain; charset=utf-8), Content-Length -> Buffer(7))
but no information about response status 200 OK
Simply:
import play.api.http.Status
if(result.status == Status.OK) {
// ...
}
Maybe I am missing here something but you have "status" on the response.
So you can do:
WS.url(url).withHeaders("Content-Type" -> "application/json").post(message).map{
case response if ( response.status == OK) => //DO SOMETHING?
}

Play 2.1 Scala SQLException Connection Timed out waiting for a free available connection

I have been working on this issue for quite a while now and I cannot find a solution...
A web app built with play framework 2.2.1 using h2 db (for dev) and a simple Model package.
I am trying to implement a REST JSON endpoint and the code works... but only once per server instance.
def createOtherModel() = Action(parse.json) {
request =>
request.body \ "name" match {
case _: JsUndefined => BadRequest(Json.obj("error" -> true,
"message" -> "Could not match name =(")).as("application/json")
case name: JsValue =>
request.body \ "value" match {
case _: JsUndefined => BadRequest(Json.obj("error" -> true,
"message" -> "Could not match value =(")).as("application/json")
case value: JsValue =>
// this breaks the secod time
val session = ThinkingSession.dummy
val json = Json.obj(
"content" -> value,
"thinkingSession" -> session.id,
)
)
Ok(Json.obj("content" -> json)).as("application/json")
}
} else {
BadRequest(Json.obj("error" -> true,
"message" -> "Name was not content =(")).as("application/json")
}
}
}
so basically I read the JSON, echo the "value" value, create a model obj and send it's id.
the ThinkingSession.dummy function does this:
def all(): List[ThinkingSession] = {
// Tried explicitly closing connection, no difference
//val conn = DB.getConnection()
//try {
// DB.withConnection { implicit conn =>
// SQL("select * from thinking_session").as(ThinkingSession.DBParser *)
// }
//} finally {
// conn.close()
//}
DB.withConnection { implicit conn =>
SQL("select * from thinking_session").as(ThinkingSession.DBParser *)
}
}
def dummy: ThinkingSession = {
(all() head)
}
So this should do a SELECT * FROM thinking_session, create a model obj list from the result and return the first out of the list.
This works fine the first time after server start but the second time I get a
play.api.Application$$anon$1: Execution exception[[SQLException: Timed out waiting for a free available connection.]]
at play.api.Application$class.handleError(Application.scala:293) ~[play_2.10.jar:2.2.1]
at play.api.DefaultApplication.handleError(Application.scala:399) [play_2.10.jar:2.2.1]
at play.core.server.netty.PlayDefaultUpstreamHandler$$anonfun$2$$anonfun$applyOrElse$3.apply(PlayDefaultUpstreamHandler.scala:261) [play_2.10.jar:2.2.1]
at play.core.server.netty.PlayDefaultUpstreamHandler$$anonfun$2$$anonfun$applyOrElse$3.apply(PlayDefaultUpstreamHandler.scala:261) [play_2.10.jar:2.2.1]
at scala.Option.map(Option.scala:145) [scala-library.jar:na]
at play.core.server.netty.PlayDefaultUpstreamHandler$$anonfun$2.applyOrElse(PlayDefaultUpstreamHandler.scala:261) [play_2.10.jar:2.2.1]
Caused by: java.sql.SQLException: Timed out waiting for a free available connection.
at com.jolbox.bonecp.DefaultConnectionStrategy.getConnectionInternal(DefaultConnectionStrategy.java:88) ~[bonecp.jar:na]
at com.jolbox.bonecp.AbstractConnectionStrategy.getConnection(AbstractConnectionStrategy.java:90) ~[bonecp.jar:na]
at com.jolbox.bonecp.BoneCP.getConnection(BoneCP.java:553) ~[bonecp.jar:na]
at com.jolbox.bonecp.BoneCPDataSource.getConnection(BoneCPDataSource.java:131) ~[bonecp.jar:na]
at play.api.db.DBApi$class.getConnection(DB.scala:67) ~[play-jdbc_2.10.jar:2.2.1]
at play.api.db.BoneCPApi.getConnection(DB.scala:276) ~[play-jdbc_2.10.jar:2.2.1]
My application.conf (db section)
db.default.driver=org.h2.Driver
db.default.url="jdbc:h2:file:database/[my_db]"
db.default.logStatements=true
db.default.idleConnectionTestPeriod=5 minutes
db.default.connectionTestStatement="SELECT 1"
db.default.maxConnectionAge=0
db.default.connectionTimeout=10000
Initially the only thing set in my config was the connection and the error occurred. I added all the other stuff while reading up on the issue on the web.
What is interesting is that when I use the h2 in memory db it works once after server start and after that it fails. when I use the h2 file system db it only works once, regardless of the server instances.
Can anyone give me some insight on this issue? Have found some stuff on bonecp problem and tried upgrading to 0.8.0-rc1 but nothing changed... I am at a loss =(
Try to set a maxConnectionAge and idle timeout
turns out the error was quite somewhere else... it was a good ol' stack overflow... have not seen one in a long time. I tried down-voting my question but it's not possible^^

Getting The entire node set is unreachable error when trying to connect to MongoDB with reactivemongo in my play2 application

When working locally I have no problem testing my application by calling:
http://localhost:9000/r/123
method:
def showSurvey(id: String) = Action {
implicit val reader = Review.ReviewReader
Async {
val cursor = reviews.find(BSONDocument("_id" -> BSONObjectID(id))).cursor[Review]
cursor.headOption.map(maybeReview =>
maybeReview.map(review => {
// fill form
Ok(views.html.review.desktopSurvey(Some(id), surveyForm.fill(SurveyForm(review.grade, review.text, REGULAR_AUTH, false, "", "")), grades))
}
).getOrElse {
//NotFound Temporary below:
val review = Review(Some(new BSONObjectID(id)), 3, "bla", Some(new DateTime()), Some(new DateTime()), Some("0.0.0.0"), ReviewStatus.NOT_CLAIMED, Some(1), Some(1L))
Ok(views.html.review.desktopSurvey(Some(id), surveyForm.fill(SurveyForm(review.grade, review.text, REGULAR_AUTH, false, "", "")), grades))
}
).recover {
case e => InternalServerError(e.getMessage())
}
}
}
But when running the app in production by doing clean compile stage and then going to the
url I get:
[error] r.c.a.MongoDBSystem - The entire node set is unreachable, is there a network problem?
I find this very strange since the plugin seems to work properly:
[info] application - ReactiveMongoPlugin successfully started with db 'blala'! Servers:
[87.238.57.140:27017]
By checking the mongodb log i found that the connection was ok but the bson sent in was invalid:
Tue May 21 11:18:11.257 [conn531] Assertion: 10307:Client Error: bad object in message: invalid bson
0xdcf361 0xd90a1b 0xd90f5c 0x75b289 0x75b3fb 0x9f4367 0x9f57e2 0x6e747a 0xdbbb7e 0x7fa22e96d9ca 0x7fa22dd14cdd
/usr/bin/mongod(_ZN5mongo15printStackTraceERSo+0x21) [0xdcf361]
/usr/bin/mongod(_ZN5mongo11msgassertedEiPKc+0x9b) [0xd90a1b]
/usr/bin/mongod() [0xd90f5c]
/usr/bin/mongod(_ZN5mongo9DbMessage9nextJsObjEv+0x249) [0x75b289]
/usr/bin/mongod(_ZN5mongo12QueryMessageC1ERNS_9DbMessageE+0x8b) [0x75b3fb]
/usr/bin/mongod() [0x9f4367]
/usr/bin/mongod(_ZN5mongo16assembleResponseERNS_7MessageERNS_10DbResponseERKNS_11HostAndPortE+0x392) [0x9f57e2]
/usr/bin/mongod(_ZN5mongo16MyMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortEPNS_9LastErrorE+0x9a) [0x6e747a]
/usr/bin/mongod(_ZN5mongo17PortMessageServer17handleIncomingMsgEPv+0x42e) [0xdbbb7e]
/lib/libpthread.so.0(+0x69ca) [0x7fa22e96d9ca]
/lib/libc.so.6(clone+0x6d) [0x7fa22dd14cdd]
Tue May 21 11:18:11.261 [conn531] AssertionException handling request, closing client connection: 10307 Client Error: bad object in message: invalid bson
This made me look at the code and the problem was that I was sending in an invalid ObjectId, in my case 123 . Somehow this error got swallowed by my application. So when there is a possibility that the ObjectId sent in might be corrupt check it with:
val maybeOID: Try[BSONObjectID] = BSONObjectID.parse(id)
if (maybeOID.isSuccess) {
// Do stuff
}