Using spray, I want to have a system that waits for some seconds between it sends two HTTP requests to specific URL, because I don't want to mess up the server's traffic for my app's auto connections. How do you do it? I can make it by putting the command in every place where it needs to pause, but I figured it's not looking cool and hard to maintain afterwords. I would love if it can be abstracted into the level of ActorSystem. Thank you!
I'm not very seasoned with Spray, but could you do a third bogus request (in between your two actual requests) with a "some seconds" timeout period? (or maybe 1/2 of "some seconds" to allow for the request retry doubling the actual time you end up waiting?) Sorry if this is unhelpful or incorrect, like I said I haven't worked with it much :/
I recommend using a TimerBasedThrottler to throttle the messages an Actor receives. Here is a quick example of what you would need
// Actor that handles making the HTTP request
class MyRequestActor extends Actor {
def receive = {
case url => makeRequest(url)
}
}
// throttles to 1 message per second
val throttler = system.actorOf(Props(new TimerBasedThrottler(new Rate(1, (1.second))))
val requestActor = system.actorOf(Props(new MyRequestActor))
// register your actor with the throttler
throttler ! SetTarget(Option(requestActor))
// send messages to the request actor through the throttler
throttler ! "some/url"
Related
I am using Play 2.6 and The twitter streaming API.
Below is how I connect to twitter using the ws library's stream() method.
The problem is that, the steam always stops after exactly 2 minutes. I tried different topics and the behavior is pretty consistent.
It seems there is a setting but I could not find where.
I am not sure it's on the play side or twitter side.
Any help is greatly appreciated.
ws.url("https://stream.twitter.com/1.1/statuses/filter.json")
.sign(OAuthCalculator(ConsumerKey(credentials._1, credentials._2), RequestToken(credentials._3, credentials._4)))
.withQueryStringParameters("track" -> topic)
.withMethod("POST")
.stream()
.map {
response => response.bodyAsSource.map(t=> {t.utf8String})
}
Play WS has default request timeout which is exactly 2 minutes by default.
Here is link to the docs:
https://www.playframework.com/documentation/2.6.x/ScalaWS#Configuring-Timeouts
So you can put in your application.conf line like
play.ws.timeout.request = 10 minutes
to specify default timeout for your all requests.
Also you can specify timeout for single request using withRequestTimeout method of WSRequest builder
/**
* Sets the maximum time you expect the request to take.
* Use Duration.Inf to set an infinite request timeout.
* Warning: a stream consumption will be interrupted when this time is reached unless Duration.Inf is set.
*/
def withRequestTimeout(timeout: Duration): WSRequest
So to disable request timeout for a sigle request you can use following code
ws.url(someurl)
.withMethod("GET")
.withRequestTimeout(Duration.Inf)
I am learning and understanding Akka actor model with some basic examples.
I created two instances for a same Actor (i.e) "helloworld1" and "helloworld2" and sending messages to that Actor. The first message is sent by instance "helloworld1" and second message is sent by "helloworld2" .
When i run the below code i get output as below
Yes you made it Hi helloworld2
Yes you made it Hi helloworld1
My question is why is it the 2nd message is processed at first? I was expecting the output as below because the first message is sent by helloworld1
Yes you made it Hi helloworld1
Yes you made it Hi helloworld2
Code for ActorApp:
import akka.actor.{ActorSystem, Props}
object ActorApp {
def main(args :Array[String]) ={
val system = ActorSystem("SimpleActorSystem")
val actorObj1 = system.actorOf(Props[HelloWorld],"helloworld1")
actorObj1 ! "Hi helloworld1"//first message sent
val actorObj2 =system.actorOf(Props[HelloWorld],"helloworld2")
actorObj2 ! "Hi helloworld2"//second message sent
}
}
Code for Actor:
import akka.actor.Actor
class HelloWorld extends Actor {
def receive = {
case dataStr:String => println("Yes you made it "+dataStr)
}
}
Akka does not guarantee that messages are delivered in the order they are sent globally. It guarantees the following:
at-most-once delivery (i.e. no guaranteed delivery, messages may be dropped)
message ordering per sender–receiver pair
So you see, the ordering is only guaranteed for any given sender-receiver pair. If you send multiple messages from "helloworld1" they will be received in the order you sent them (however messages from other sources may come in between).
See the documentation for an in-depth explanation.
So I have a client server based program, where the client will send a request to the server, the server will do a computation and response. This is done via ask.
Specifically the client will receive a message from the client app and send call ask
val response = ask(actorRef, SessionMessage(token, message)).mapTo[ResponseMessage]
The server will receive it like so
val response = sessionMessage.message match {
case message: message1 =>
ask(actorSet.actor1,message)
case message: message2 =>
ask(actorSet.actor2,message)
Where the actorset is literally a set of the different actors.
I then collect the result and send back to the sender
val responseResult = response.mapTo[ResponseMessage]
responseResult pipeTo sender
The problem I'm running into is that for some of the requests, the database query can take a while (5-10 minutes) and when the query completes it sends to dead letters and I get a dissociation and it is unable to associate again and sends to dead letters.
I thought that because it took so long, that the sender would time out (or specifically the sender reference) so I stored the sender reference as a val, and confirmed that by doing this I the sender reference was lost. However, as soon as as the query finishes and I pipe it to the correct sender, it dissociates. Even other queries that take a minute or so don't seem to suffer this problem, only ones that last for a few minutes dissociate and I need to restart the server or the server will keep sending to dead letters.
Even if I do a onComplete then send on success or do an Await.result, the same issue occurs, as soon as it tries to send the message (after completion) the server dissociates and sends to dead letters.
I'm very much at lost as to why this is happening.
The problem you are into is that ask itself has a timeout, which is separate from a timeout you might specify in Await.result. The full signature to ask is:
def ask (actorRef: ActorRef, message: Any)(implicit timeout: Timeout): Future[Any]
This means that if you did not manually provide a value for timeout and did not define an implicit yourself, you must be inheriting one via one of your imports.
To extend the timeout for your particular ask, simply call it with one:
ask(actorRef, SessionMessage(token, message))(15.minutes).mapTo[ResponseMessage]
or if this applies to all asks in scope, declare your own implicit:
implicit val timeout = Timeout(15.minutes)
When I get one message from a non akka client through TCP socket, I need to reply as three messages. In the following sample given below, only the first one goes through properly to the sender (the TCP client which is non AKKA). The rest of the two goes to dead letter. Any idea? Thanks in advance.
object TcpExample {
def main(args: Array[String]): Unit = {
val system = ActorSystem("some-system")
val tcpConsumer = system.actorOf(Props[TcpConsumer])
}
class TcpConsumer extends Consumer {
def endpointUri = "mina2:tcp://localhost:6200?textline=true"
def receive = {
case msg: CamelMessage => {
sender ! msg.bodyAs[String]
sender ! msg.bodyAs[String] // This goes to dead letter
sender ! msg.bodyAs[String] // This goes to dead letter
}
case msg: Failure => sender ! msg
}
}
Without knowing too much about the internals of the akka/camel integration, let me try and demonstrate what's happening here. First, as I mentioned in my comment, the sender in your actor does not directly refer to the TCP client that is on the other side of the system. It's lower level than that; it's whatever ActorRef sent your Consumer the CamelMessage in the first place. So what actor is that? Let me try and explain what I think is happening.
When you set up tcp based a camel consumer, based on the endpointUri there will be a piece of code (from Camel) that is going to bind to the host and port from the endpointUri.
When a new connection request comes in (based on an external client opening a connection to that socket), some sort of actor is probably spun up to handle that individual connection. So there will be 1-n "connection handler" actor instances matching the number of open connections.
When a message comes inbound, it more than likely goes through that connection handler actor. From there, it is either being sent to your consumer via ask (?), or another short lived actor is being spun up to handle that individual message.
Either way, next stop is your consumer, where it's receive function gets hit with a CamelMessage representing the payload from the message sent from the remove client. When this happens, the actors sender is still whatever sent the message in step 3.
Your consumer will now send a message back to the sender and then from there it will eventually be routed back to the connection handler for that connection. In there, it will write back to the socket, in a conversational state. One message in, one message out.
I think your problem is that you are breaking the "one in, one out" paradigm here. When you get your CamelMessage, you are only supposed to respond to that message once, which will evantually trickle back up to the TCP client on the other end of the socket. I don't think the framework expects another response, and that's why you see deadletters for the other two responses.
So this begs the question, what scenario do you have that requires a "1 in, 3 out" paradigm vs the expected "1 in, one out" one that the framework seems to expect?
streaming data out of play, is quite easy.
here's a quick example of how I intend to do it (please let me know if i'm doing it wrong):
def getRandomStream = Action { implicit req =>
import scala.util.Random
import scala.concurrent.{blocking, ExecutionContext}
import ExecutionContext.Implicits.global
def getSomeRandomFutures: List[Future[String]] = {
for {
i <- (1 to 10).toList
r = Random.nextInt(30000)
} yield Future {
blocking {
Thread.sleep(r)
}
s"after $r ms. index: $i.\n"
}
}
val enumerator = Concurrent.unicast[Array[Byte]] {
(channel: Concurrent.Channel[Array[Byte]]) => {
getSomeRandomFutures.foreach {
_.onComplete {
case Success(x: String) => channel.push(x.getBytes("utf-8"))
case Failure(t) => channel.push(t.getMessage)
}
}
//following future will close the connection
Future {
blocking {
Thread.sleep(30000)
}
}.onComplete {
case Success(_) => channel.eofAndEnd()
case Failure(t) => channel.end(t)
}
}
}
new Status(200).chunked(enumerator).as("text/plain;charset=UTF-8")
}
now, if you get served by this action, you'll get something like:
after 1757 ms. index: 10.
after 3772 ms. index: 3.
after 4282 ms. index: 6.
after 4788 ms. index: 8.
after 10842 ms. index: 7.
after 12225 ms. index: 4.
after 14085 ms. index: 9.
after 17110 ms. index: 1.
after 21213 ms. index: 2.
after 21516 ms. index: 5.
where every line is received after the random time has passed.
now, imagine I want to preserve this simple example when streaming data from the server to the client, but I also want to support full streaming of data from the client to the server.
So, lets say i'm implementing a new BodyParser that parses the input into a List[Future[String]]. this means, that now, my Action could look like something like this:
def getParsedStream = Action(myBodyParser) { implicit req =>
val xs: List[Future[String]] = req.body
val enumerator = Concurrent.unicast[Array[Byte]] {
(channel: Concurrent.Channel[Array[Byte]]) => {
xs.foreach {
_.onComplete {
case Success(x: String) => channel.push(x.getBytes("utf-8"))
case Failure(t) => channel.push(t.getMessage)
}
}
//again, following future will close the connection
Future.sequence(xs).onComplete {
case Success(_) => channel.eofAndEnd()
case Failure(t) => channel.end(t)
}
}
}
new Status(200).chunked(enumerator).as("text/plain;charset=UTF-8")
}
but this is still not what I wanted to achieve. in this case, I’ll get the body from the request only after the request was finished, and all the data was uploaded to the server. but I want to start serving request as I go. a simple demonstration, would be to echo any received line back to the user, while keeping the connection alive.
so here's my current thoughts:
what if my BodyParser would return an Enumerator[String] instead of List[Future[String]]?
in this case, I could simply do the following:
def getParsedStream = Action(myBodyParser) { implicit req =>
new Status(200).chunked(req.body).as("text/plain;charset=UTF-8")
}
so now, i'm facing the problem of how to implement such a BodyParser.
being more precise as to what exactly I need, well:
I need to receive chunks of data to parse as a string, where every string ends in a newline \n (may contain multiple lines though...). every "chunk of lines" would be processed by some (irrelevant to this question) computation, which would yield a String, or better, a Future[String], since this computation may take some time. the resulted strings of this computation, should be sent to the user as they are ready, much like the random example above. and this should happen simultaneously while more data is being sent.
I have looked into several resources trying to achieve it, but was unsuccessful so far.
e.g. scalaQuery play iteratees -> it seems like this guy is doing something similar to what I want to do, but I couldn't translate it into a usable example. (and the differences from play2.0 to play2.2 API doesn't help...)
So, to sum it up: Is this the right approach (considering I don't want to use WebSockets)? and if so, how do I implement such a BodyParser?
EDIT:
I have just stumble upon a note on the play documentation regarding this issue, saying:
Note: It is also possible to achieve the same kind of live
communication the other way around by using an infinite HTTP request
handled by a custom BodyParser that receives chunks of input data, but
that is far more complicated.
so, i'm not giving up, now that I know for sure this is achievable.
What you want to do isn't quite possible in Play.
The problem is that Play can't start sending a response until it has completely received the request. So you can either receive the request in its entirety and then send a response, as you have been doing, or you can process requests as you receive them (in a custom BodyParser), but you still can't reply until you've received the request in its entirety (which is what the note in the documentation was alluding to - although you can send a response in a different connection).
To see why, note that an Action is fundamentally a (RequestHeader) => Iteratee[Array[Byte], SimpleResult]. At any time, an Iteratee is in one of three states - Done, Cont, or Error. It can only accept more data if it's in the Cont state, but it can only return a value when it's in the Done state. Since that return value is a SimpleResult (i.e, our response), this means there's a hard cut off from receiving data to sending data.
According to this answer, the HTTP standard does allow a response before the request is complete, but most browsers don't honor the spec, and in any case Play doesn't support it, as explained above.
The simplest way to implement full-duplex communication in Play is with WebSockets, but we've ruled that out. If server resource usage is the main reason for the change, you could try parsing your data with play.api.mvc.BodyParsers.parse.temporaryFile, which will save the data to a temporary file, or play.api.mvc.BodyParsers.parse.rawBuffer, which will overflow to a temporary file if the request is too large.
Otherwise, I can't see a sane way to do this using Play, so you may want to look at using another web server.
"Streaming data in and out simultaneously on a single HTTP connection in play"
I haven't finished reading all of your question, nor the code, but what you're asking to do isn't available in HTTP. That has nothing to do with Play.
When you make a web request, you open a socket to a web server and send "GET /file.html HTTP/1.1\n[optional headers]\n[more headers]\n\n"
You get a response after (and only after) you have completed your request (optionally including a request body as part of the request). When and only when the request and response are finished, in HTTP 1.1 (but not 1.0) you can make a new request on the same socket (in http 1.0 you open a new socket).
It's possible for the response to "hang" ... this is how web chats work. The server just sits there, hanging onto the open socket, not sending a response until someone sends you a message. The persistent connection to the web server eventually provides a response when/if you receive a chat message.
Similarly, the request can "hang." You can start to send your request data to the server, wait a bit, and then complete the request when you receive additional user input. This mechanism provides better performance than continually creating new http requests on each user input. A server can interpret this stream of data as a stream of distinct inputs, even though that wasn't necessarily the initial intention of the HTTP spec.
HTTP does not support a mechanism to receive part of a request, then send part of a response, then receive more of a request. It's just not in the spec. Once you've begun to receive a response, the only way to send additional information to the server is to use another HTTP request. You can use one that's already open in parallel, or you can open a new one, or you can complete the first request/response and issue an additional request on the same socket (in 1.1).
If you must have asynchronous io on a single socket connection, you might want to consider a different protocol other than HTTP.