I am using Akka-HTTP to return a response in form of String [30 - 40 MB].
When I deploy my Akka-HTTP server and make a request to fetch the data from some URI then every time it gets stuck after several MB's and stop fetching the complete response.
Is there any way that I return my whole large response without getting stuck in between.
HttpResponse(StatusCodes.OK, entity = myLargeResponseAsString)
Thanks
Maybe this could help:
path("yourpath") {
get {
complete {
val str2 = scala.io.Source.fromFile("/tmp/t.log", "UTF8").mkString
val str = Source.single(ByteString(str2))
HttpResponse(entity = HttpEntity.Chunked.fromData(ContentTypes.`text/plain(UTF-8)`, str))
}
}
Related
I am working on a web application where the user's connection times out after a specific time (say 20 seconds). For long running requests I have to return a default message ("your request is under process") and then send an email to the user with the actual result.
I couldn't do this with spring web because I didn't know how to specify a timeout in the controller (with customized messages per request) and at the same time let other requests come through and be processed too. That's why I used spring web-flux which has a timeout operator for both Mono and Flux types.
To make the requested process run in a different thread, I have used Sinks. One to receive requests and one to publish the results. My problem is that the response sink can only return one result and subsequent calls to the URL returns an empty response. For example the first call to /reactive/getUser/123456789 returns the user object but subsequent calls return empty.
I'm not sure if the problem is with the Sink I have used or with how I am getting data from it. In the sample code I have used responseSink.asFlux().next() but I have also tried .single(), .toMono(), .take(1). to no avail. I get the same result.
#RequestMapping("/reactive")
#RestController
class SampleController #Autowired constructor(private val externalService: ExternalService) {
private val requestSink = Sinks.many().multicast().onBackpressureBuffer<String>()
private val responseSink = Sinks.many().multicast().onBackpressureBuffer<AppUser>()
init {
requestSink.asFlux()
.map { phoneNumber -> externalService.findByIdOrNull(phoneNumber) }
.doOnNext {
if (it != null) {
responseSink.tryEmitNext(it)
} else {
responseSink.tryEmitError(Throwable("didn't find a value for that phone number"))
}
}
.subscribe()
}
#GetMapping("/getUser/{phoneNumber}")
fun getUser(#PathVariable phoneNumber: String): Mono<String> {
requestSink.tryEmitNext(phoneNumber)
return responseSink.asFlux()
.next()
.map { it.toString() }
.timeout(Duration.ofSeconds(20), Mono.just("processing your request"))
}
}
I am new to Akka Streams and gRPC, I am trying to build an endpoint where client sends a single request and the server sends multiple responses.
This is my protobuf
syntax = "proto3";
option java_multiple_files = true;
option java_package = "customer.service.proto";
service CustomerService {
rpc CreateCustomer(CustomerRequest) returns (stream CustomerResponse) {}
}
message CustomerRequest {
string customerId = 1;
string customerName = 2;
}
message CustomerResponse {
enum Status {
No_Customer = 0;
Creating_Customer = 1;
Customer_Created = 2;
}
string customerId = 1;
Status status = 2;
}
I am trying to achieve this by sending customer request then the server will first check and respond No_Customer then it will send Creating_Customer and finally server will say Customer_Created.
I have no idea where to start for it implementation, looked for hours but still clueless, I will be very thankful if anyone can point me in the right direction.
The place to start is the Akka gRPC documentation and, in particular, the service WalkThrough. It is pretty straightforward to get the samples working in a clean project.
The relevant server sample method is this:
override def itKeepsReplying(in: HelloRequest): Source[HelloReply, NotUsed] = {
println(s"sayHello to ${in.name} with stream of chars...")
Source(s"Hello, ${in.name}".toList).map(character => HelloReply(character.toString))
}
The problem is now to create a Source that returns the right results, but that depends on how you are planning to implement the server so it is difficult to answer. Check the Akka Streams documentation for various options.
The client code is simpler, just call runForeach on the Source that gets returned by CreateCustomer as in the sample:
def runStreamingReplyExample(): Unit = {
val responseStream = client.itKeepsReplying(HelloRequest("Alice"))
val done: Future[Done] =
responseStream.runForeach(reply => println(s"got streaming reply: ${reply.message}"))
done.onComplete {
case Success(_) =>
println("streamingReply done")
case Failure(e) =>
println(s"Error streamingReply: $e")
}
}
Does anyone have experienced about slow response from using graphQL ?
This is my code in resolver:
getActiveCaseWithActiveProcess(){
console.log ("getActiveCaseWithActiveProcess");
var result = [];
var activeElements = ActiveElements.find({
type:"signal",
$or:[
{signalRef:"start-process"},
{signalRef:"start-task"},
{signalRef:"close-case"}
]
},{limit:200}).fetch();
for (var AE of activeElements){
var checkAECount = ActiveElements.find({caseId:AE['caseId']}).count();
if (checkAECount <= 3){
console.log ('caseId: ' + AE['caseId']);
var checkExistInResult = result.filter(function (obj) {
return obj.caseId === AE['caseId'];
})[0];
if (checkExistInResult == null){
result.push({
caseId: AE['caseId'],
caseStart: AE['createdDate']
});
}
}
}
console.log("loaded successfully");
return result;
}
I have a huge data from my collection actually. Approximately 20000 records. However when I load this, the response is too slow and it can repeat to reload by itself which makes the response is even longer.
I20160812-04:07:25.968(0)? caseId: CASE-0000000284,
I20160812-04:07:26.890(0)? caseId: CASE-0000000285
I20160812-04:07:28.200(0)? caseId: CASE-0000000285
I20160812-04:07:28.214(0)? getActiveCaseWithActiveProcess
I20160812-04:07:28.219(0)? caseId: CASE-0000000194
I20160812-04:07:29.261(0)? caseId: CASE-0000000197
As you notice from my attachment above, at this time(20160812-04:07:28.214) the server repeats to load from the beginning again, and that's why the response will take longer.
This is not always happening. It happens when the server loads slowly. When the server loads fast. Everything just runs smoothly.
Not really enough information to answer that question here, but my guess would be that it has nothing to do with GraphQL. I think your client just cancels the request and makes another one because the first one timed out. You can find out if that happens by logging requests to your server before they're passed to GraphQL.
I'm using akka-http to make a request to a http service which sends back chunked response. This is how the relevant bit of code looks like:
val httpRequest: HttpRequest = //build the request
val request = Http().singleRequest(httpRequest)
request.flatMap { response =>
response.entity.dataBytes.runForeach { chunk =>
println("-----")
println(chunk.utf8String)
}
}
and the output produced in the command line looks something like this:
-----
{"data":
-----
"some text"}
-----
{"data":
-----
"this is a longer
-----
text"}
-----
{"data": "txt"}
-----
...
The logical piece of data - a json in this case ends with an end of line symbol \r\n, but the problem is, that the json doesn't always fit in a single http response chunk as clearly visible in the example above.
My question is - how do I concatenate the incoming chunked data into full jsons so that the resulting container type would still remain either Source[Out,M1] or Flow[In,Out,M2]? I'd like to follow the idealogy of akka-stream.
UPDATE: It's worth mentioning also, that the response is endless and the aggregation must be done in real time
Found a solution:
val request: HttpRequest = //build the request
request.flatMap { response =>
response.entity.dataBytes.scan("")((acc, curr) => if (acc.contains("\r\n")) curr.utf8String else acc + curr.utf8String)
.filter(_.contains("\r\n"))
.runForeach { json =>
println("-----")
println(json)
}
}
The akka stream documentation has an entry in the cookbook for this very problem: "Parsing lines from a stream of ByteString". Their solution is quite verbose but can also handle the situation where a single chunk can contain multiple lines. This seems more robust since the chunk size could change to be big enough to handle multiple json messages.
response.entity.dataBytes
.via(Framing.delimiter(ByteString("\n"), maximumFrameLength = 8096))
.mapAsyncUnordered(Runtime.getRuntime.availableProcessors()) { data =>
if (response.status == OK) {
val event: Future[Event] = Unmarshal(data).to[Event]
event.foreach(x => log.debug("Received event: {}.", x))
event.map(Right(_))
} else {
Future.successful(data.utf8String)
.map(Left(_))
}
}
The only requirement is you know the maximum size of one record. If you start with something small, the default behavior is to fail if the record is larger than the limit. You can set it to truncate instead of failing, but piece of a JSON makes no sense.
I am trying to build a service that grab some pages from another web service and process the content and return results to users. I am using Play 2.2.3 Scala.
val aas = WS.url("http://localhost/").withRequestTimeout(1000).withQueryString(("mid", mid), ("t", txt)).get
val result = aas.map {
response =>
(response.json \ "status").asOpt[Int].map {
st => status = st
}
(response.json \ "msg").asOpt[String].map {
txt => msg = txt
}
}
val rs1 = Await.result(result, 5 seconds)
if (rs1.isDefined) {
Ok("good")
}
The problem is that the service will wait 5 seconds to return "good" even the WS request takes 100 ms. I also cannot set Await time to 100ms because the other web service I am requesting may take between 100ms to 1 second to respond.
My question is: is there a way to process and serve the results as soon as they are ready instead of wait a fixed amount of time?
#wingedsubmariner already provided the answer. Since there is no code example, I will just post what it should be:
def wb = Action.async{ request =>
val aas = WS.url("http://localhost/").withRequestTimeout(1000).get
aas.map(response =>{
Ok("responded")
})
}
Now you don't need to wait until the WS to respond and then decide what to do. You can just tell play to do something when it responds.