Openshift - java.lang.OutOfMemoryError: unable to create new native thread" - threadpool

I have hosted my app on Openshift PAAS. My app does not create thread explicity. I have been getting following error
m.sun.jersey.spi.container.ContainerResponse mapMappableContainerException
SEVERE: The exception contained within MappableContainerException could not be mapped to a response, re-throwing to the HTTP container
java.lang.OutOfMemoryError: unable to create new native thread
How do I resolve the error? I don't have the access to change ulimit as I am using hosted server.

Generally, you face “java.lang.OutOfMemoryError: Unable to create new native thread” whenever JVM is asking a new thread from the OS. Whenever the underlying OS cannot allocate a new native thread, this OutOfMemoryError will be thrown. The exact limit for native threads is platform dependent.
But in general, the situation causing the java.lang.OutOfMemoryError: Unable to create new native thread goes through the following phases:
A new Java thread is requested by application running inside the JVM
JVM native code proxies the request to create a new native thread to the OS
OS tries to create a new native thread which requires memory to be allocated to the thread
The OS will refuse native memory allocation either because the 32-bit Java process size has depleted its memory address space e.g. (2-4) GB process size limit has been hit or the virtual memory of the OS has been fully depleted
The java.lang.OutOfMemoryError: Unable to create new native thread error is thrown.
More often than not, the limits on new native threads hit by the OutOfMemoryError indicate a programming error. When your application is spawning thousands of threads then chances are that something has gone terribly wrong – there are not many applications out there which would benefit from such a vast amount of threads

Related

Catalina-utility-1 and Catalina-utility-2 excessive memory allocation

What are these Catalina-utility-X threads and why do they burn through so much heap? They consume 20x more memory than my worker thread, which I was trying to debug because I thought it was memory-intensive.
This snapshot is taken on a tomcat instance on my Eclipse IDE, which is receiving zero http traffic - only doing the work on that worker thread (or so I thought).

Why is Visual VM showing a thread as parked when it is getting scheduled?

I have created a custom Execution Context (FixedThreadPool with 100 workers) in Scala to run IO Bound Futures. I am making db calls to Cassandra in these futures.
I have made the context available to the DB related Futures. In the log messages I see that these threads are getting executed in the specified thread pool.
However Visual VM tells a different story.
This shows that the threads have a CPU utilisation of 0%.
What am I missing here ?

Wildfly 8.2 stop incoming connections suddenly

We have a wildfly 8.2 app server which allocates 6GB of server RAM. Sometime due to havey transaction count wildfly has stop receiving incoming connections. But when I check server (not app server, it is our VM ) memory, it uses 4GB of RAM. Then I checked Wildfly app server's heap memory it did not use at least 25% of allocate heap size. Why is that? When I restart wildfly App server, All things work normally and when it comes that kind of load, above scenario happen again
Try to increase the connection-limit as suggested in this SO question
You can Dump HTTP requests as given here
Also, are you getting any errors in your console? Please post them as well.

Is it possible to handle memcached memory geting overflow?

I have a serious problem my memcached memory is overflow and server is getting down.
So how to handle memcached, If memcached memory is getting full then it will just throw error msg, not set the memcached anymore.
memcached is distributed cache system and can be works on different servers. What is your server? is it couchebase, is it elastic cache of AWS? you can use memcache on many different server and providers and when you are creating those servers you need to configure them and set the size of memory you want memcache to use. for example in the company I am working, the test environment uses couchbase but the live uses Amazon Elastic cache.
Memcached uses LRU (least recently used) algorithm to insert the new object into the memory if the table is full. You should not have problem because of full memory and handling memcached. The problem can raise from the full memory but not with this reason that memcached cannot handle it. Exceptions and other problem can be in other part of the system which is quite normal if your memory is full. if you configure the server correctly memcache usually does not throw exception.
Is the server that runs memcached same as the server that runs your application? Memcached can be put on another server and in this way you can prevent the memory to become full.

Meteor crashing

My Meteor app is crashing with the following error:
Unexpected mongo exit code null. Restarting.
=> Exited from signal: SIGKILL
/home/ron/.meteor/packages/meteor-tool/.1.1.3.4sddkj++os.linux.x86_64+web.browser+web.cordova/mt-os.linux.x86_64/dev_bundle/lib/node_modules/fibers/future.js:245
throw(ex);
^
Error: Unable to allocate ArrayBuffer.
This is followed by a call-stack trace.
What is causing this?
Thanks!
This error is probably caused by your operating environment. If its not able to allocate an ArrayBuffer it may be that you don't have enough RAM or some other service is blocking meteor from allocating memory.
This error may occur on the smallest DigitalOcean droplet if that's what you're using.
It's generally recommended you have 1 GB of free ram for Meteor to work properly in development mode.
Something you could use is a swapfile to increase your ram.
Real RAM memory could be replaced with virtual memory but won't be so fast memory... in linux this SO feature is achieved using a swap partition. In windows is using a paging file. Weirdly you can emulate this feature in the linux world using swapspace (or create a traditional swap partition)
sudo apt-get install swapspace
Whatever option you choose will create swap for you and it will help you to start up your meteor app!!!
Just be aware that this will be a more slower experience than real RAM but definitely will work