WSVR0652W: The size of thread pool "WebContainer" has reached 80 percent of its maximum.
I got the above error.can anyone tell me when we will get the above message?
This is an informational message. The Application Server does emit messages when
the size of the WebContainer thread pool approaches its maximum.
Related
We are experiencing an issue with Celery+SQS, using the gevent or the eventlet pools.
Even if we have a large amount of messages in the queue, and the concurrency set to 100, prefetch multiplier set to 5 (so the total prefetch_count reported is 500) celery will receive 10 messages from the queue (the max allowed by SQS) and wait for all of them to finish before attempting to receive again (from AWS we see either 10 or 0 messages in flight).
We do not see the same behavior using the prefork pool.
Our tasks are I/O bound, so using prefork would be inefficient, is there any way to increase the amount of messages pulled from the queue to keep all the greenlet busy?
[1] 2022-01-18 21:56:10,280 ERROR [org.apa.cam.pro.err.DefaultErrorHandler] (Camel (camel-1) thread #9 - KafkaProducer[test]) Failed delivery for (MessageId: 95835510BC9E9B2-0000000000134315 on ExchangeId: 95835510BC9E9B2-0000000000134315). Exhausted after delivery attempt: 1 caught: org.apache.kafka.common.errors.TimeoutException: Expiring 1 record(s) for test-0:121924 ms has passed since batch creation
[1]
[1] Message History (complete message history is disabled)
[1] ---------------------------------------------------------------------------------------------------------------------------------------
[1] RouteId ProcessorId Processor Elapsed (ms)
[1] [route1 ] [route1 ] [from[netty://udp://0.0.0.0:8080?receiveBufferSize=65536&sync=false] ] [ 125320]
[1] ...
[1] [route1 ] [to1 ] [kafka:test?brokers=10.99.155.100:9092&producerBatchSize=0 ] [ 0]
[1]
[1] Stacktrace
[1] ---------------------------------------------------------------------------------------------------------------------------------------
[1] : org.apache.kafka.common.errors.TimeoutException: Expiring 1 record(s) for test-0:121924 ms has passed since batch creation
Here's the flow for my project
External Service ---> Netty
Netty ---> Kafka(consumer)
Kafka(producer) ---> processing events
1 and 2 are running in one Kubernetes pod and 3 is running in a separate pod.
I have encountered TimeoutException at the beginning saying like:
org.apache.kafka.common.errors.TimeoutException: Expiring 20 record(s) for test-0:121924 ms has passed since batch creation
I searched online and found a couple of potential solutions
Kafka Producer error Expiring 10 record(s) for TOPIC:XXXXXX: 6686 ms has passed since batch creation plus linger time
Based on the suggestion, I have done:
make the timeout bigger, double the default value
make the batch size to 0, which will not send events in batch and keeps the memory usage low.
Unfortunately I still encounter the error due to memory is used up.
Does anyone know how to solve it?
Thanks!
There are several things to take into account here.
You are not showing up what your throughput is, you have to take into account that value and if your broker on 10.99.155.100:9092 is able to process such load.
Did you check 10.99.155.100 during the time of the transfer? The fact that Kafka can potentially process hundreds of thousands of messages per second doesn't mean that you can do it on any hardware.
So, having said that, the timeout is the first to come to my mind, but in your case you have 2 minutes and still you are timing out, for me, this sounds more like a problem in your broker and not on your producer.
To understand the issue, basically, you are getting your mouth full faster than you can swallow, by the time push a message the broker is not able to acknowledge on time (in this case, 2 minutes)
What things you can do here:
Check the broker performance for the given load Change your
delivery.timeout.ms to an acceptable value, I guess you have SLAs
to attach to Increase your retry backoff timer (retry.backoff.ms)
Do not put the batch size as 0, this will try a live push to the
broker, which in case seems not possible for the load. Make sure your
max.block.ms is set correctly Change to bigger batches (even if this
increases latency), but not too big, you need to sit down, check how
many records you are pushing and allocate them correctly.
Now, some rules:
delivery.timeout.ms must be bigger than the sum of
request.timeout.ms and linger.ms All the above are impacted by
the batch.size If you don't have so many rows, but those rows are
huge! then control the max.request.size
So, to summarize, your properties to change are the following:
delivery.timeout.ms, request.timeout.ms, linger.mx, max.request.size
Assuming the hardware is good and also assuming that you are not sending more than you should, those should do the trick
Our project has Microsoft Service Bus (on-prem ) running on Windows 2012 R2 servers for message processing.
When sending messages to service bus topic above the size limit (say 10 mb ) , services bus shows processing error – throws socket timeout exception.
Just wanted to know ,
if anyone has worked with sending messages (say > 10 MBs ) to Service Bus Topics . Would appreciate any suggested approach on how to handle this.
Also is there a way to increase the service bus timeout configuration or message size limit settings on Service Bus Topics either through Powershell cmds or Service Bus Explorer.
Service Bus queues support a maximum message size of 256 Kb (the header, which includes the standard and custom application properties, can have a maximum size of 64 Kb).
There is no limit on the number of messages held in a queue but there is a cap on the total size of the messages held by a queue. This queue size is defined at creation time, with an upper limit of 5 GB.
Are you asking about sending a message which is of size 10 MB? Service Bus doesn't allow that large message. For Premium, the maximum message size is 1 MB, and for Standard, it's 256 KB as #Ana said.
Also is there a way to increase the service bus timeout configuration
or message size limit settings?
Yes, there is a possibility to handle time-to-live property of messages which can be configured either at the time of Queue/Subscription creation or while sending Individual message. Refer to set Time to live for Queue as well as message.
Also is there a way to increase message size limit settings?
No, as the maximum size is 1 MB (May be increased by Azure in the future).
To answer this "Can we Send messages (say > 10 MBs ) to Service Bus Topics".
Now as of today, the updated answer will be YES: The Premium tier of Service Bus, enabling Message size up to 100 MB. Where as Standard is up to 256 KB as of today.
How to enabling large messages support for an existing queue (or topic)
Recommended:
While 100 MB message payloads are supported, it's recommended to keep the message payloads as small as possible to ensure reliable performance from the Service Bus namespace.
The Premium tier is recommended for production scenarios.
If Thread: 100, Rampup: 1 and Loop count: 1 is the configuration, how will jmeter start sending requests to the server?
Request will be sent 1 req/sec or all requests will be sent all at once to server?
JMeter will send requests as fast as it can, to wit:
It will start all threads (virtual users) you define in Thread Group within the ramp-up period (in your case - 100 threads in 1 second)
Each thread (virtual user) will start executing Samplers which are present in the Thread Group upside down (or according to the Logic Controllers)
When there are no more samplers to execute or loops to iterate the thread will be shut down
When there are no more active threads left - JMeter test will end.
With regards to requests per second - it mostly depends on your application response time, i.e.
if you have 100 virtual users and response time is 1 second - you will get 100 requests/second
if you have 100 virtual users and response time is 2 seconds - you will get 50 requests/second
if you have 100 virtual users and response time is 500 milliseconds - you will get 200 requests/second
etc.
I would recommend increasing (and decreasing) the load gradually, this way you will be able to correlate increasing load with increasing throughput/response time/number of errors, etc. while releasing all threads at once will not tell you the full story (unless you're doing a form of spike testing, in this case consider using Synchronizing Timer)
JMeter's ramp-up period set as 1 means to start all 100 threads in 1 second.
This isn't recommended settings as describe below
The ramp-up period tells JMeter how long to take to "ramp-up" to the full number of threads chosen. If 10 threads are used, and the ramp-up period is 100 seconds, then JMeter will take 100 seconds to get all 10 threads up and running. Each thread will start 10 (100/10) seconds after the previous thread was begun. If there are 30 threads and a ramp-up period of 120 seconds, then each successive thread will be delayed by 4 seconds.
Ramp-up needs to be long enough to avoid too large a work-load at the start of a test, and short enough that the last threads start running before the first ones finish (unless one wants that to happen).
Start with Ramp-up = number of threads and adjust up or down as needed.
See also Can i set ramp up period 0 in JMeter?
bear in mind that with low rampup and many threads, you may be limited by local resources, so your results may be a measurement of client capability rather than server.
We are keep getting work manager thread limit error in weblogic server log. Though our app is working fine, i would like to know what is the issue here and what kind of impact it can have in future to our app. Since this just the POC we are kind of ignoring it
java.lang.RuntimeException: MaxThreads constraint 'HARVESTER_WM' queue for workManager 'HARVESTER_WM' exceeded maximum capacity of '8000' elements. Max threads constraint count is set to 1