I build server with akka-http and akka-stream but it lost some request in 2K+ request. What I miss something or my understand for akka is wrong.
this is my code
implicit val actorRef = ActorSystem("system", testConf)
implicit val materializer = ActorMaterializer(ActorMaterializerSettings(actorRef).withInputBuffer(initialSize = 4, maxSize = 16))
implicit val requestTimeout = Timeout(8 seconds)
def response(req: HttpRequest): Future[Response] = {
val connectionFlow: Flow[HttpRequest, HttpResponse, Future[Http.OutgoingConnection]] =
Http().outgoingConnection(host = "url").async
for {
res <- Source.single(req.copy(uri = s"${req.uri.path}?${req.uri.rawQueryString.get}")).via(connectionFlow).runWith(Sink.head)
data <- res.entity.toStrict(5 second)
} yield (data.getData().decodeString("UTF-8"), res.status.intValue())
}
Thank you.
Most likely there were either timeout or server-side errors in first part of your for-comprehension therefore you got only successful responses in res.
I recommend to create flow/graph that processes your requests with withSupervisionStrategy in your materializer so you can see what exactly went wrong. Details of implementation depend on your business logic.
Related
I have a controller that sends a chunked response:
def streamDatase2t(query:String): Action[AnyContent] = Action.async {
req =>
serivce.getIterator(query).map(res => {
Ok.chunked(Source.apply(res))
})
}
When I try to inspect the returned content in the controller spec I get an exception:
"return 200 response with the content of the iterator" in {
when(serivce.getIterator
(Matchers.any[Request.DatasetLoad],
Matchers.any[ResponseFormat], Matchers.any[Int]))
.thenReturn(Future.successful(new FakeIterable(List("One", "Two", "Three").iterator)))
val fakeRequest = FakeRequest.apply("GET", s"/data")
val result = Helpers.route(fakeApp, fakeRequest).get
checkStatus(result, OK)
contentAsString(result) // <-- exception here !
}
Exception:
NoMaterializer cannot materialize
java.lang.UnsupportedOperationException: NoMaterializer cannot materialize
at play.api.test.NoMaterializer$.materialize(Helpers.scala:732)
at akka.stream.scaladsl.RunnableGraph.run(Flow.scala:629)
at akka.stream.scaladsl.Source.runWith(Source.scala:106)
at akka.stream.scaladsl.Source.runFold(Source.scala:117)
at play.api.http.HttpEntity.consumeData(HttpEntity.scala:49)
at play.api.http.HttpEntity.consumeData$(HttpEntity.scala:48)
at play.api.http.HttpEntity$Chunked.consumeData(HttpEntity.scala:117)
at play.api.test.ResultExtractors.contentAsBytes(Helpers.scala:381)
at play.api.test.ResultExtractors.contentAsBytes$(Helpers.scala:379)
at play.api.test.Helpers$.contentAsBytes(Helpers.scala:676)
As the Exception states NoMaterializer cannot materialize you may need to add a Materializer:
implicit lazy val mat = ActorMaterializer()
implicit lazy val ec = instanceOf[ExecutionContext]
contentAsString has NoMaterializer as the default argument
def contentAsString(of: Future[Result])(implicit timeout: Timeout, mat: Materializer = NoMaterializer): String
NoMaterializer just throws UnsupportedOperationException for everything so try providing your own
implicit val actorSystem = ActorSystem("test")
implicit val materializer = ActorMaterializer()
play-scala-streaming-example demonstrates how we might write a test for streaming controller.
Addressing the comment, consider the following two routes which illustrate the difference between a strict and non-strict (chunked, streamed) body
def nonStrictBody = Action {
val source = Source.apply(List("woo", "h", "oo"))
Ok.chunked(source)
}
def strictBody = Action {
Ok("woohoo")
}
When calling contentAsString on a strict body, then materializer will not be used, hence NoMaterializer is sufficient
In 99% of cases, when running tests against the result body, you don't
actually need a materializer since it's a strict body. So, rather than
always requiring an implicit materializer, we use one if provided,
otherwise we have a default one that simply throws an exception if
used.
However when calling contentAsString on a chunked or streamed body, as it is the case in the nonStrictBody route, then we need to provide a proper Materializer.
I'am trying to implement Akka http client for post request where Authorization will be given in header from postman. I am not able to authorize, following is my code
def main(args: Array[String]) {
implicit val system: ActorSystem = ActorSystem()
implicit val materializer: ActorMaterializer = ActorMaterializer()
implicit val executionContext: ExecutionContextExecutor = system.dispatcher
val requestHeader = scala.collection.immutable.Seq(RawHeader("Authorization", "admin"))
val requestHandler: HttpRequest => HttpResponse = {
case HttpRequest(POST, Uri.Path("/GetTrackerData"), requestHeader, entity, _) =>
val chunk = Unmarshal(entity).to[DeviceLocationData]
val deviceLocationData = Await.result(chunk, 1.second)
val responseArray = "Authorized"
HttpResponse(entity = HttpEntity(ContentTypes.`application/json`, responseArray)
)
case r: HttpRequest =>
println(r.uri.toString())
r.discardEntityBytes() // important to drain incoming HTTP Entity stream
HttpResponse(404, entity = "Unknown resource!")
}
val bindingFuture = Http().bindAndHandleSync(requestHandler, "0.0.0.0", 7070)
println(s"iot engine api live at 0.0.0.0:7070")
sys.addShutdownHook({
println("ShutdownHook called")
bindingFuture
.flatMap(_.unbind()) // trigger unbinding from the port
.onComplete(_ => system.terminate()) // and shutdown when done
})
}
Whatever value I give from postman. It serves the request. What I am skipping ?
My use case is that result showed be displayed only after authorization
You are pattern matching on HttpRequest.
The requestHeader you use there is not the one you specified earlier but will be the headers from the HttpRequest itself.
One way to resolve it could be checking for the values in the headers:
case HttpRequest(HttpMethods.POST, Uri.Path("/GetTrackerData"), headers, entity, _)
if (headers.exists(h => h.name == "Authorization" && h.value == "admin")) =>
Here is one problem:
case HttpRequest(POST, Uri.Path("/GetTrackerData"), requestHeader, entity, _) =>
This does not match the current value of requestHeader, it creates a new value requestHeader containing the current headers of the HttpRequest. So this match will not check the Authorization field of the header. You will have to do this manually.
More generally, it would be worth looking at the Route support in Akka HTTP as a cleaner and more powerful way of implementing a server.
I have a very simple Akka WebSocket server that pushes lines from a file to a connected client with an interval of 400ms per line. Everything works fine, except for the fact that the web server seems to buffer messages for about a minute before broadcasting them.
So when a client connects, I see at the server end that every 400ms a line is read and pushed to the Sink, but on the client side I get nothing for a minute and then a burst of about 150 messages (corresponding to a minute of messages).
Is there a setting that I'm overlooking?
object WebsocketServer extends App {
implicit val actorSystem = ActorSystem("WebsocketServer")
implicit val materializer = ActorMaterializer()
implicit val executionContext = actorSystem.dispatcher
val file = Paths.get("websocket-server/src/main/resources/EURUSD.txt")
val fileSource =
FileIO.fromPath(file)
.via(Framing.delimiter(ByteString("\n"), Int.MaxValue))
val delayedSource: Source[Strict, Future[IOResult]] =
fileSource
.map { line =>
Thread.sleep(400)
println(line.utf8String)
TextMessage(line.utf8String)
}
def route = path("") {
extractUpgradeToWebSocket { upgrade =>
complete(upgrade.handleMessagesWithSinkSource(
Sink.ignore,
delayedSource)
)
}
}
val bindingFuture = Http().bindAndHandle(route, "localhost", 8080)
bindingFuture.onComplete {
case Success(binding) ⇒
println(s"Server is listening on ws://localhost:8080")
case Failure(e) ⇒
println(s"Binding failed with ${e.getMessage}")
actorSystem.terminate()
}
}
So the approach with Thread.sleep(400) was wrong. I should've used the .throttle mechanic on sources:
val delayedSource: Source[Strict, Future[IOResult]] =
fileSource
.throttle(elements = 1, per = 400.millis)
.map { line =>
println(line.utf8String)
TextMessage(line.utf8String)
}
This fixed the issue.
This is my code for the server written using Akka framework:
case class Sentence(data: String)
case class RawTriples(triples: List[String])
trait Protocols extends DefaultJsonProtocol {
implicit val sentenceRequestFormat = jsonFormat1(Sentence)
implicit val rawTriplesFormat = jsonFormat1(RawTriples)
}
trait Service extends Protocols {
implicit val system: ActorSystem
implicit def executor: ExecutionContextExecutor
implicit val materializer: Materializer
val openie = new OpenIE
def config: Config
val logger: LoggingAdapter
lazy val ipApiConnectionFlow: Flow[HttpRequest, HttpResponse, Any] =
Http().outgoingConnection(config.getString("services.ip-api.host"), config.getInt("services.ip-api.port"))
def ipApiRequest(request: HttpRequest): Future[HttpResponse] = Source.single(request).via(ipApiConnectionFlow).runWith(Sink.head)
val routes = {
logRequestResult("akka-http-microservice") {
pathPrefix("openie") {
post {
decodeRequest{
entity(as[Sentence]){ sentence =>
complete {
var rawTriples = openie.extract(sentence.data)
val resp: MutableList[String] = MutableList()
for(rtrip <- rawTriples){
resp += (rtrip.toString())
}
val response: List[String] = resp.toList
println(response)
response
}
}
}
}
}
}
}
}
object AkkaHttpMicroservice extends App with Service {
override implicit val system = ActorSystem()
override implicit val executor = system.dispatcher
override implicit val materializer = ActorMaterializer()
override val config = ConfigFactory.load()
override val logger = Logging(system, getClass)
Http().bindAndHandle(routes, config.getString("http.interface"), config.getInt("http.port"))
}
The server accepts a POST request containing a sentence and returns a json array in return. It works fine but if I am making requests to it too frequently using parallelized code, then it gives 500 Internal server error. I wanted to know is there any parameter which I can set in the server to avoid that (number of ready threads for accepting requests etc).
In log files, the error is logged as:
[ERROR] [05/31/2017 11:48:38.110]
[default-akka.actor.default-dispatcher-6]
[akka.actor.ActorSystemImpl(default)] Error during processing of
request: 'null'. Completing with 500 Internal Server Error response.
The doc on the bindAndHandle method shows what you want:
/**
* Convenience method which starts a new HTTP server at the given endpoint and uses the given `handler`
* [[akka.stream.scaladsl.Flow]] for processing all incoming connections.
*
* The number of concurrently accepted connections can be configured by overriding
* the `akka.http.server.max-connections` setting. Please see the documentation in the reference.conf for more
* information about what kind of guarantees to expect.
*
* To configure additional settings for a server started using this method,
* use the `akka.http.server` config section or pass in a [[akka.http.scaladsl.settings.ServerSettings]] explicitly.
*/
akka.http.server.max-connections is probably what you want. As the doc suggests, you can also dig deeper into the akka.http.server config section.
Add following in application.conf file
akka.http {
server {
server-header = akka-http/${akka.http.version}
idle-timeout = infinite
request-timeout = infinite
}
}
I'm trying to build event sourced service with REST interface using scala. I somewhat new to scala, although I'm familiar with functional programming (haskell at beginner level).
So I've build persistent actor and view without major problems. The idea of actors is quite simple I think.
object Main extends App {
val system = ActorSystem("HelloSystem")
val systemActor = system.actorOf(Props[SystemActor], name = "systemactor")
val trajectoryView = system.actorOf(Props[TrajectoryView], name = "trajectoryView")
var datas = List()
val processData = ProcessData(0, List(1,2,3), Coordinates(50, 50))
implicit val timeout = Timeout(5 seconds)
def intialDatas(): List[ProcessData] =
(for (i <- 1 to 3) yield ProcessData(i, List(1,2,3), Coordinates(50 + i, 50 + i)))(collection.breakOut)
val command = RegisterProcessCommand(3, this.intialDatas())
val id = Await.result(systemActor ? command, timeout.duration).asInstanceOf[String]
println(id)
systemActor ! MoveProcessCommand(4, ProcessData(4, List(3,4,5), Coordinates(54, 54)), id)
val processes = Await.result(systemActor ? "get", timeout.duration).asInstanceOf[Set[Process]]
println(processes)
implicit val json4sFormats = DefaultFormats
println(write(processes))
println("*****************")
systemActor ! "print"
val getTrajectoryCommand = GetTrajectoryCommand(id)
Thread.sleep(10000)
trajectoryView ! "print"
// val trajectory = Await.result(trajectoryView ? getTrajectoryCommand, timeout.duration).asInstanceOf[ListBuffer[Coordinates]]
println("******* TRAJECTORY *********")
trajectoryView ! "print"
// println(trajectory)
system.shutdown()
}
I've been able to create a script for playing with actor that I've created.
I've read the tutorials for spray routing, but I've been unable to grasp what exactly should I do to provide REST interface for actors that I've created.
object Boot extends App{
implicit val system = ActorSystem("example")
val systemActor = system.actorOf(Props[SystemActor], name = "systemactor")
val trajectoryView = system.actorOf(Props[TrajectoryView], name = "trajectoryView")
val service = system.actorOf(Props[ProcessesService], "processes-rest-service")
implicit val timeout = Timeout(5 seconds)
IO(Http) ? Http.Bind(service, interface = "localhost", port = 8080)
}
And a service
class ProcessesService(systemActor: ActorRef) extends Actor with HttpService {
def actorRefFactory = context
def receive = runRoute(route)
val json4sFormats = DefaultFormats
implicit val timeout = Timeout(5 seconds)
val route = path("processes") {
get {
respondWithMediaType(`application/json`) {
complete {
write(Await.result(systemActor ? "get", timeout.duration).asInstanceOf[Set[Process]])
}
}
}
}
}
I think I need to somehow pass actorRef for SystemActor to this ProcessesService, but I'm not sure how. Also I'm not sure how should I return a response to the request. I understand that I need to somehow pass the "get" message to SystemActor through ActorRef and then serialize the answer to json, but I don't know how to do that.
I would appreciate help!
In spray you can complete routes with a Future.
You should be able to do something like
complete { systemActor ? "get" }
Json serialization is a separate issue.
Oh, your question is vague. Yes you need to be able to reference an actor within your routes. You could just import the val from boot where you define it. They're just Scala variables so where you put them is up to you.