I have a GET endpoint that sometimes returns a JSON payload larger than 10MB, thus giving me error code 500 as follows:
Execution failed due to configuration error:
Integration response of reported length 10507522 is larger than allowed maximum of 10485760 bytes.
I was aware of the 10MB payload quota, so I enabled Content Encoding in the Settings and added the HTTP header Accept-Encoding:gzip to the request.
Now, payloads that were 7-8MB uncompressed are being sent with a size of about 350KB, so the compression works. Still, payloads over 10MB continue to give that same error.
Why does it keep giving me this error? Is it checking payload size before compression? How can I fix this?
Related
How to increase the header size limit in haproxy, currently the default settings allowed only 8KB.
Due to this we are facing 400 error at client side.
localhost haproxy[21502]: xx.xx.xx.xx:xxxx [xx/xxx/xxxx:xx:xx:xx.xx] www-http www-http/ -1/-1/-1/-1/0 400 187 - - PR-- 411/23/0/0/0 0/0 ""
You can set the tune.maxrewrite to a higher value.
It is strongly recommended that the header should be lower then the 8KB.
Here a link which shows the header limits on different servers Maximum HTTP request header size defaults compared across web servers
I use Kafka version 2.2.0cp2 through Rest Proxy (in the Docker container). I need the consumer to always read only one message.
I set the value max.poll.records=1 in the file /etc/kafka/consumer.properties as follows:
consumer.max.poll.records=1
OR:
max.poll.records=1
It had no effect.
Setting this value in other configs also did not give any result.
So consumer.properties is not read from the REST Proxy
Assuming consumer properties can be changed, the kafka-rest container env-var would be KAFKA_REST_CONSUMER_MAX_POLL_RECORDS, but that setting only controls the inner poll loop of the Proxy server, not the returned amount of data to the HTTP client...
There would have to be a limit flag given to the API, which does not exist - https://docs.confluent.io/current/kafka-rest/api.html#get--consumers-(string-group_name)-instances-(string-instance)-records
I don't see any consumer poll setting mentioned in the below link
https://docs.confluent.io/current/kafka-rest/config.html
But if you know the average message size you can pass max_bytes as below to control record size
GET
/consumers/testgroup/instances/my_consumer/records?timeout=3000&max_bytes=300000
HTTP/1.1
max_bytes:
The maximum number of bytes of unencoded keys and values that should
be included in the response. This provides approximate control over
the size of responses and the amount of memory required to store the
decoded response. The actual limit will be the minimum of this setting
and the server-side configuration consumer.request.max.bytes. Default
is unlimited
I am reading slim 3 docs and found that it does read/send the response body text in chunks of 4096 bytes:
responseChunkSize Size of each chunk read from the Response body when
sending to the browser. (Default: 4096)
What is the point of doing it so? Wouldn't it better to send the response body at once? Would this imply a small overhead?
During sending response to client browser, content length of response body may or may not be available.
In both cases, responseChunkSize settings is used as number of bytes to read from body until it reaches end of file. If content length is known and it is less or equal than responseChunkSize, then it only takes one iteration to read body's content.
By reading and output response in smaller chunk, browser does not wait too long to get first byte. Reading big chunk is slower and may require larger memory consumption so browser will likely get first byte longer than smaller chunk.
Reading the Wikipedia page, it seems that there is some overlap between the two. Which is the best practice when uploaded max file size is exceded?
413 is the answer.
400 - Bad Request
The request could not be understood by the server due to malformed syntax. The client SHOULD NOT repeat the request without modifications.
413 Request Entity Too Large / Payload Too Large
The server is refusing to process a request because the request entity is larger than the server is willing or able to process. [...]
Sources:
400 status spec
413 status spec
Does this value only affect incoming traffic? If i set this value to say 4MB on zookeeper server as well as zookeeper client and I start my client, will I still get data > 4MB when I do a request for a path /abc/asyncMultiMap/subs. If /subs has data greater than 4MB is the server going to break it up in chunks <= 4MB and send it in pieces to the client?
I am using zookeeper 3.4.6 on both client (via vertx-zookeeper) and server. I see errors on clients where it complains that packet length is greater than 4MB.
java.io.IOException: Packet len4194374 is out of range!
at org.apache.zookeeper.ClientCnxnSocket.readLength(ClientCnxnSocket.java:112) ~[zookeeper-3.4.6.jar:3.4.6-1569965]
"This is a server-side setting"
This statement is incorrect, jute.maxbuffer is evaluated on client as well by Record implementing classes that receive InputArchive. Each time a field is read and stored into an InputArchive the value is checked against jute.maxbuffer. Eg ClientCnxnSocket.readConnectResult
I investigated it in ZK 3.4
There is no chunking in the response.
This is a server-side setting. You will get this error if the entirety of the response is greater than the jute.maxbuffer setting. This response limit includes the list of children of znodes as well so even if subs does not have a lot of data but has enough children such that their length of their paths exceeds the max buffer size you will get the error.