I am trying to make a singleRequest from Akka using this code:
val request = HttpRequest(
method = HttpMethods.GET,
uri = "url"
)
val responseFuture: Future[HttpResponse] = Http().singleRequest(request)
val entityFuture: Future[HttpEntity.Strict] = responseFuture.flatMap(response => response.entity.toStrict(2.seconds))
entityFuture.map(entity => entity.data.utf8String)
However when I request a big json string I get the following exception.
akka.http.scaladsl.model.EntityStreamException: HTTP chunk size exceeds the configured limit of 1048576 bytes
How do I configure this, I am not using Akka typed(I think), just this:
implicit val system = ActorSystem()
implicit val materializer = ActorMaterializer()
import system.dispatcher
You can use the following configuration to increase the request size:
akka.http.parsing.max-chunk-size = 16m
Related
I have a controller that sends a chunked response:
def streamDatase2t(query:String): Action[AnyContent] = Action.async {
req =>
serivce.getIterator(query).map(res => {
Ok.chunked(Source.apply(res))
})
}
When I try to inspect the returned content in the controller spec I get an exception:
"return 200 response with the content of the iterator" in {
when(serivce.getIterator
(Matchers.any[Request.DatasetLoad],
Matchers.any[ResponseFormat], Matchers.any[Int]))
.thenReturn(Future.successful(new FakeIterable(List("One", "Two", "Three").iterator)))
val fakeRequest = FakeRequest.apply("GET", s"/data")
val result = Helpers.route(fakeApp, fakeRequest).get
checkStatus(result, OK)
contentAsString(result) // <-- exception here !
}
Exception:
NoMaterializer cannot materialize
java.lang.UnsupportedOperationException: NoMaterializer cannot materialize
at play.api.test.NoMaterializer$.materialize(Helpers.scala:732)
at akka.stream.scaladsl.RunnableGraph.run(Flow.scala:629)
at akka.stream.scaladsl.Source.runWith(Source.scala:106)
at akka.stream.scaladsl.Source.runFold(Source.scala:117)
at play.api.http.HttpEntity.consumeData(HttpEntity.scala:49)
at play.api.http.HttpEntity.consumeData$(HttpEntity.scala:48)
at play.api.http.HttpEntity$Chunked.consumeData(HttpEntity.scala:117)
at play.api.test.ResultExtractors.contentAsBytes(Helpers.scala:381)
at play.api.test.ResultExtractors.contentAsBytes$(Helpers.scala:379)
at play.api.test.Helpers$.contentAsBytes(Helpers.scala:676)
As the Exception states NoMaterializer cannot materialize you may need to add a Materializer:
implicit lazy val mat = ActorMaterializer()
implicit lazy val ec = instanceOf[ExecutionContext]
contentAsString has NoMaterializer as the default argument
def contentAsString(of: Future[Result])(implicit timeout: Timeout, mat: Materializer = NoMaterializer): String
NoMaterializer just throws UnsupportedOperationException for everything so try providing your own
implicit val actorSystem = ActorSystem("test")
implicit val materializer = ActorMaterializer()
play-scala-streaming-example demonstrates how we might write a test for streaming controller.
Addressing the comment, consider the following two routes which illustrate the difference between a strict and non-strict (chunked, streamed) body
def nonStrictBody = Action {
val source = Source.apply(List("woo", "h", "oo"))
Ok.chunked(source)
}
def strictBody = Action {
Ok("woohoo")
}
When calling contentAsString on a strict body, then materializer will not be used, hence NoMaterializer is sufficient
In 99% of cases, when running tests against the result body, you don't
actually need a materializer since it's a strict body. So, rather than
always requiring an implicit materializer, we use one if provided,
otherwise we have a default one that simply throws an exception if
used.
However when calling contentAsString on a chunked or streamed body, as it is the case in the nonStrictBody route, then we need to provide a proper Materializer.
I have a method which send picture from client to CDN throught FormData. Code:
def uploadToCDN(formData: Multipart.FormData): Future[HttpResponse] = {
implicit val system = ActorSystem()
implicit val materializer = ActorMaterializer()
implicit val executionContext = system.dispatcher
Http().singleRequest(
HttpRequest(
method = HttpMethods.POST,
uri = "http://cdn.example.com",
entity = formData.toEntity(),
protocol = HttpProtocols.`HTTP/1.1`))
}
How I can add "secret_key": "12345678" to FormData which I receive from the client?
Multipart.FormData is basically made up of its parts. To join two FormDatas you need to concatenate the formdata parts and create a new instance of FormData:
val newFormData =
Multipart.FormData(
Source.single(Multipart.FormData.BodyPart("secret_key", "12345678"))
.concat(originalFormData.parts)
)
See also the Scaladocs of Multipart.FormData.
I have been trying to create an application that runs recursive task (crawler) with akka streams and QPID message broker.
What I have noticed is that separate parts of the graph alone perform quite good, but when connected together performance drops down significantly.
Here is statistics for the graph running on my local machine:
sending messages to message queue can achieve > 1700 msg/sec;
making HTTP requests approx. 70 req/sec;
whole graph, including reading
messages from queue 2-4 items/sec;
Source code for the pipeline can be found here:
https://gist.github.com/volisoft/3617824b16a3f3b6e01c933a8bdf8049
The pipeline is straightforward:
def main(args: Array[String]): Unit = {
startBroker()
val queueName = "amqp-conn-it-spec-simple-queue-" + System.currentTimeMillis()
val queueDeclaration = QueueDeclaration(queueName)
val in = AmqpSource(
NamedQueueSourceSettings(AmqpConnectionDetails("localhost", 5672, Some(AmqpCredentials("guest", "guest"))), queueName)
.withDeclarations(queueDeclaration),
bufferSize = 1028
).map(_.bytes.utf8String).log(":in")
val out = AmqpSink.simple(
AmqpSinkSettings(AmqpConnectionDetails("localhost", 5672, Some(AmqpCredentials("guest", "guest"))))
.withRoutingKey(queueName).withDeclarations(queueDeclaration))
val urlsSink = Flow[String].map(ByteString(_)).to(out)
val g = RunnableGraph.fromGraph(GraphDSL.create(in, urlsSink)((_,_)){ implicit b => (in, urlsSink0) =>
import GraphDSL.Implicits._
val pool = Http().superPool[String]()(materializer).log(":pool")
val download: Flow[String, Document, NotUsed] = Flow[String]
.map(url => (HttpRequest(method = HttpMethods.GET, Uri(url)), url) )
.via(pool)
.mapAsyncUnordered(8){ case (Success(response: HttpResponse), url) => parse(response, url)}
val filter = Flow[String].filter(notVisited).log(":filter")
val save = Flow[String].map(saveVisited)
val extractLinks: Flow[Document, String, NotUsed] = Flow[Document].mapConcat(document => getUrls(document))
in ~> save ~> download ~> extractLinks ~> filter ~> urlsSink0
ClosedShape
})
g.run()
Source.single(rootUrl).map(s => ByteString(s)).runWith(out)
}
How can this code be optimized to increase performance?
I build server with akka-http and akka-stream but it lost some request in 2K+ request. What I miss something or my understand for akka is wrong.
this is my code
implicit val actorRef = ActorSystem("system", testConf)
implicit val materializer = ActorMaterializer(ActorMaterializerSettings(actorRef).withInputBuffer(initialSize = 4, maxSize = 16))
implicit val requestTimeout = Timeout(8 seconds)
def response(req: HttpRequest): Future[Response] = {
val connectionFlow: Flow[HttpRequest, HttpResponse, Future[Http.OutgoingConnection]] =
Http().outgoingConnection(host = "url").async
for {
res <- Source.single(req.copy(uri = s"${req.uri.path}?${req.uri.rawQueryString.get}")).via(connectionFlow).runWith(Sink.head)
data <- res.entity.toStrict(5 second)
} yield (data.getData().decodeString("UTF-8"), res.status.intValue())
}
Thank you.
Most likely there were either timeout or server-side errors in first part of your for-comprehension therefore you got only successful responses in res.
I recommend to create flow/graph that processes your requests with withSupervisionStrategy in your materializer so you can see what exactly went wrong. Details of implementation depend on your business logic.
I'm trying out Akka Streams and here is a short snippet that I have:
override def main(args: Array[String]) {
val filePath = "/Users/joe/Softwares/data/FoodFacts.csv"//args(0)
val file = new File(filePath)
println(file.getAbsolutePath)
// read 1MB of file as a stream
val fileSource = SynchronousFileSource(file, 1 * 1024 * 1024)
val shaFlow = fileSource.map(chunk => {
println(s"the string obtained is ${chunk.toString}")
})
shaFlow.to(Sink.foreach(println(_))).run // fails with a null pointer
def sha256(s: String) = {
val messageDigest = MessageDigest.getInstance("SHA-256")
messageDigest.digest(s.getBytes("UTF-8"))
}
}
When I ran this snippet, I get:
Exception in thread "main" java.lang.NullPointerException
at akka.stream.scaladsl.RunnableGraph.run(Flow.scala:365)
at com.test.api.consumer.DataScienceBoot$.main(DataScienceBoot.scala:30)
at com.test.api.consumer.DataScienceBoot.main(DataScienceBoot.scala)
It seems to me that it is not fileSource is just empty? Why is this? Any ideas? The FoodFacts.csv if 40MB in size and all I'm trying to do is to create a 1MB stream of data!
Even using the defaultChunkSize of 8192 did not work!
Well 1.0 is deprecated. And if you can, use 2.x.
When I try with 2.0.1 version by using FileIO.fromFile(file) instead of SynchronousFileSource, it is a compile failure with message fails with null pointer. This was simply because it didnt have ActorMaterializer in scope. Including it, makes it work:
object TImpl extends App {
import java.io.File
implicit val system = ActorSystem("Sys")
implicit val materializer = ActorMaterializer()
val file = new File("somefile.csv")
val fileSource = FileIO.fromFile(file,1 * 1024 * 1024 )
val shaFlow: Source[String, Future[Long]] = fileSource.map(chunk => {
s"the string obtained is ${chunk.toString()}"
})
shaFlow.runForeach(println(_))
}
This works for file of any size. For more information on configuration of dispatcher, refer here.