I'm trying to send a get request to my localhost using Akka HTTP and I get the following exception:
EntityStreamSizeException: actual entity size (Some(10166731700)) exceeded content length limit (8388608 bytes)! You can configure this by setting akka.http.[server|client].parsing.max-content-length or calling HttpEntity.withSizeLimit before materializing the dataBytes stream.)
Basically, the file on my localhost which I'm trying to request is very large.
I tried to solve it by using withoutSizeLimit, but it's not working:
val request = Get("http://localhost:8080/oa-ok.ntriples")
val limitedRequest = request.withEntity(request.entity.withoutSizeLimit())
val responseFuture = Http().singleRequest(limitedRequest)
I also tried to use withSizeLimit and it didn't help. Any ideas?
Use withoutSizeLimit on the response entity, not the request entity. For example:
val responseFuture: Future[HttpResponse] =
Http().singleRequest(HttpRequest(uri = "http://localhost:8080/oa-ok.ntriples"))
val responseSource: Source[ByteString, NotUsed] =
Source.fromFuture(responseFuture)
.flatMapConcat(_entity.withoutSizeLimit.dataBytes)
responseSource is an Akka Streams Source of the response entity.
Related
When running a subscription source from the Alpakka PubSub library I received possible encoded data.
#Singleton
class Consumer #Inject()(config: Configuration, credentialsService: google.creds.Service)(implicit actorSystem: ActorSystem) {
implicit val m: ActorMaterializer = ActorMaterializer.create(actorSystem)
val logger = Logger(this.getClass)
val subName: String = config.get[String]("google.pubsub.subname")
val credentials: Credentials = credentialsService.getCredentials
val pubSubConfig = PubSubConfig(credentials.projectId, credentials.clientEmail, credentials.privateKey)
val subSource: Source[ReceivedMessage, NotUsed] = GooglePubSub.subscribe(subName, pubSubConfig)
val ackSink: Sink[AcknowledgeRequest, Future[Done]] = GooglePubSub.acknowledge(subName, pubSubConfig)
val computeGraph = Flow[ReceivedMessage].map {
x =>
logger.info(x.message.data)
x
}
val ackGraph = Flow.fromFunction((msgs: Seq[ReceivedMessage]) => AcknowledgeRequest(msgs.map(_.ackId).toList))
subSource
.via(computeGraph)
.groupedWithin(10, 5.minutes)
.via(ackGraph)
.to(ackSink)
.run()
}
I publish the message from the PubSub console. I am expecting my test message to appear however when publishing test I receive dGVzdA==. Is this an expected result? I have had issues with importing the private key and this might be a result of it?
The consumer is bound eagerly with Guice.
Data that is received over REST apis will be base64 encoded. My guess would be that the Alpakka Pub/Sub library which uses the REST APIs is not properly decoding the received data. It looks like they also have a library that uses the GRPC Pub/Sub client as the underlying layer which may not suffer from this defect? You can also use the Cloud Pub/Sub Java client library from Scala directly.
I have to send following curl POST request in my akka code where it is sending both json payload and file in the request:
curl \
-F "payload=</tmp/upload_file_payload.json" \
-F "file=#/tmp/file.pdf" \
-v https://host/api
I am trying to implement the same in akka-http, but not sure exactly how to do it. I found some example here and here, but it is not working as it is, so I have written following code, which also have some error, but seems like I am close:
val httpEntity = HttpEntity(MediaTypes.`application/octet-stream`, file, 100000)
// val httpEntity = HttpEntity.fromPath(ContentType.apply(MediaTypes.`application/octet-stream`), Paths.get(file.getAbsolutePath))
val fileFormData = Multipart.FormData.BodyPart.Strict("file", httpEntity, Map.empty)
val jsonFormData = Multipart.FormData.BodyPart.Strict("payload", payload, Map.empty)
// Multipart.FormData.Strict(scala.collection.immutable.Seq(jsonFormData, fileFormData)).toEntity()
val entity = Multipart.FormData( Source(List(jsonFormData, fileFormData))).toEntity()
val httpRequest = HttpRequest(HttpMethods.POST, uri = uri, entity = entity)
But this code is not compiling.
In between, when I made the code compile, I was getting error:
411 Length Requered
I tried added Content-Length header with request but no avail.
Finally after looking at the akka-http code and more attempts, finally I got this working as following, explanation in comments:
val httpEntity = HttpEntity(MediaTypes.`application/octet-stream`, file, 100000).toStrict(10.seconds)(mat) // I had to convert this into strict: which adds Content-Length and tells it is not streaming
val fileFormData = Multipart.FormData.BodyPart.Strict("file", Await.result(httpEntity, 10.seconds), Map.empty) // used Await here to get httpEntity from Future, this might be made better
val jsonFormData = Multipart.FormData.BodyPart.Strict("payload", payload, Map.empty)
val entity = Multipart.FormData(jsonFormData, fileFormData).toEntity() // Corrected this signature
val httpRequest = HttpRequest(HttpMethods.POST, uri = uri, entity = entity)
To avoid 411 error I had to make FormData strict, whcih itself adds required Content-Length header.
I build server with akka-http and akka-stream but it lost some request in 2K+ request. What I miss something or my understand for akka is wrong.
this is my code
implicit val actorRef = ActorSystem("system", testConf)
implicit val materializer = ActorMaterializer(ActorMaterializerSettings(actorRef).withInputBuffer(initialSize = 4, maxSize = 16))
implicit val requestTimeout = Timeout(8 seconds)
def response(req: HttpRequest): Future[Response] = {
val connectionFlow: Flow[HttpRequest, HttpResponse, Future[Http.OutgoingConnection]] =
Http().outgoingConnection(host = "url").async
for {
res <- Source.single(req.copy(uri = s"${req.uri.path}?${req.uri.rawQueryString.get}")).via(connectionFlow).runWith(Sink.head)
data <- res.entity.toStrict(5 second)
} yield (data.getData().decodeString("UTF-8"), res.status.intValue())
}
Thank you.
Most likely there were either timeout or server-side errors in first part of your for-comprehension therefore you got only successful responses in res.
I recommend to create flow/graph that processes your requests with withSupervisionStrategy in your materializer so you can see what exactly went wrong. Details of implementation depend on your business logic.
Im trying to parse some data from an API
I have a recursion method that calling to this method
def getJsonValue( url: (String)): JsValue = {
val builder = new com.ning.http.client.AsyncHttpClientConfig.Builder()
val client = new play.api.libs.ws.ning.NingWSClient(builder.build())
val newUrl = url.replace("\"", "").replace("|", "%7C").trim
val response: Future[WSResponse] = client.url(newUrl).get()
Await.result(response, Duration.create(10, "seconds")).json
}
Everything is working well but after 128 method calls i'm getting this warning
WARNING: You are creating too many HashedWheelTimer instances. HashedWheelTimer is a shared resource that must be reused across the application, so that only a few instances are created.
After about 20 More calls im getting this exception
23:24:57.425 [main] ERROR com.ning.http.client.AsyncHttpClient - Unable to instantiate provider com.ning.http.client.providers.netty.NettyAsyncHttpProvider. Trying other providers.
23:24:57.438 [main] ERROR com.ning.http.client.AsyncHttpClient - org.jboss.netty.channel.ChannelException: Failed to create a selector.
Questions
1.Im assuming that the connections didnt closed ?? and therefore i can't create new connections.
2.What will be the correct and the safe way to create those HTTP calls
Had the same problem.
Found 2 interesting solutions:
make sure you are not creating tons of clients with closing them
the threadPool you are using may be causing this.
My piece of code (commenting that line of code solved, I'm now testing several configurations):
private[this] def withClient(block: NingWSClient => WSResponse): Try[WSResponse] = {
val config = new NingAsyncHttpClientConfigBuilder().build()
val clientConfig = new AsyncHttpClientConfig.Builder(config)
// .setExecutorService(new ThreadPoolExecutor(5, 15, 30L, TimeUnit.SECONDS, new SynchronousQueue[Runnable]))
.build()
val client = new NingWSClient(clientConfig)
val result = Try(block(client))
client.close()
result
}
for avoiding this you can use different provider.
private AsyncHttpProvider httpProvider =new ApacheAsyncHttpProvider(config);
private AsyncHttpClient asyncHttpClient = new AsyncHttpClient(httpProvider,config);
I ran into this same problem. Before you call your recursive method, you should create builder and client and pass client to the recursive method, as well as getJsonValue. This is what getJsonValue should look like:
def getJsonValue(url: String, client: NingWSClient): JsValue = {
val builder = new com.ning.http.client.AsyncHttpClientConfig.Builder()
val client = new play.api.libs.ws.ning.NingWSClient(builder.build())
val newUrl = url.replace("\"", "").replace("|", "%7C").trim
val response: Future[WSResponse] = client.url(newUrl).get()
Await.result(response, Duration.create(10, "seconds")).json
}
I'm using Spray API(spray-client) to hit an external URL and I'm getting gzipped HttpResponse. How do I unzip this HttpResponse to get its entity(json, in my case)?
val future: Future[HttpResponse] = (IO(Http) ? Get(uri)).mapTo[HttpResponse]
val response = Await.result(future, Duration.inf)
val json = response.entity
Here, json is gzipped. How do I unzip it?
You need to use pipelining and the decode directive. Like in this example.
Modifying that example your code would look something like this:
val pipeline: HttpRequest => Future[String] = (
sendReceive
~> decode(Gzip)
~> unmarshal[String]
)
val response: Future[String] =
pipeline(Get(uri))
You can then do Await on the response if you don't want the benefits of Futures.
On a side note you can use spray-json and create an object for your response and then unmarshal the http response directly into a case class without having to deal with the json.