Heroku memory leak with Play2 scala - scala

Was doing some stretch (ab) test to my 1 heroku dyno and dev database with 20 connections limit.
During the calls (that access database with squeryl the heap allocation is increasing causing R14 (memory more than 512MB))
I cannot seem to reproduce the problem (at that levels at least locally).
Is there any way to get heroku heap dump and analyze it to get some clue?
Is there any known issues with play2, scala, squeryl and heroku memory leak?
Update
If i do System.gc at the end of the controller everything seems to be fine and slower ofc...I create a lot of object at that call but shouldn't heroku's JVM take care of gc? Also if i schedule gc call periodically don't free memory

There's a great article for troubleshooting memory issues on Heroku:
https://devcenter.heroku.com/articles/java-memory-issues
In your case, you can add the GC flags to JAVA_OPTS to see memory details. I'd suggest the following flags:
heroku config:add JAVA_OPTS="-Xmx384m -Xss512k -XX:+UseCompressedOops -XX:+PrintGCDetails -XX:+PrintHeapAtGC -XX:+PrintGCDateStamps"
There's also a simple java agent that you can add to your process if you want a little more info from JMX about your memory. You can also take a look at monitoring addons like New Relic if you want to go into more depth, but I think you should be fine with the flags and java agent.

I had this issue as well, and answered it here.
I had the same issue. Heroku is telling you the machine is running out
of memory, not the Java VM. There is actually a bug in the Heroku Play
2.2 deployment, the startup script reads java_opts, not JAVA_OPTS.
I fixed it by setting both:
heroku config:add java_opts='-Xmx384m -Xms384m -Xss512k -XX:+UseCompressedOops'
heroku config:add JAVA_OPTS='-Xmx384m -Xms384m -Xss512k -XX:+UseCompressedOops'
I also had to set -Xms otherwise I got an error saying the min and max
were incompatible. I guess Play2.2 was using a default higher than
384m.

Related

Running Scala module with databricks-connect

I've tried to follow the instructions here to set up databricks-connect with IntelliJ. My understanding is that I can run code from the IDE and it will run on the databricks cluster.
I added the jar directory from the miniconda environment and moved it above all of the maven dependencies in File -> Project Structure...
However I think I did something wrong. When I tried to run my module I got the following error:
21/07/17 22:44:24 ERROR SparkContext: Error initializing SparkContext.
java.lang.IllegalArgumentException: System memory 259522560 must be at least 471859200. Please increase heap size using the --driver-memory option or spark.driver.memory in Spark configuration.
at org.apache.spark.memory.UnifiedMemoryManager$.getMaxMemory(UnifiedMemoryManager.scala:221)
at org.apache.spark.memory.UnifiedMemoryManager$.apply(UnifiedMemoryManager.scala:201)
at org.apache.spark.SparkEnv$.create(SparkEnv.scala:413)
at org.apache.spark.SparkEnv$.createDriverEnv(SparkEnv.scala:262)
at org.apache.spark.SparkContext.createSparkEnv(SparkContext.scala:291)
at org.apache.spark.SparkContext.<init>(SparkContext.scala:495)
at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2834)
at org.apache.spark.sql.SparkSession$Builder.$anonfun$getOrCreate$2(SparkSession.scala:1016)
at scala.Option.getOrElse(Option.scala:189)
at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:1010)
at com.*.sitecomStreaming.sitecomStreaming$.main(sitecomStreaming.scala:184)
at com.*.sitecomStreaming.sitecomStreaming.main(sitecomStreaming.scala)
The system memory being 259 gb makes me think it's trying to run locally on my laptop instead of the dbx cluster? I'm not sure if this is correct and what I can do to get this up and running properly...
Any help is appreciated!
The driver in the databricks-connect is always running locally - only the executors are running in the cloud. Also, this reported memory is in the bytes, so 259522560 is ~256Mb - you can increase it using the option that it reports.
P.S. But if you're using structured streaming, then yes - it's a known limitation of databricks-connect.

IBM Cloud Dev Ops Memory Discrepency between CF App command and SSH top command

I am having an issue with my deployed Flask app on IBM Cloud where each API call is resulting in an incremental level of memory used according to the cf app command.
Since I cannot share my code my question here is related to my debugging checks. Specifically I'm wondering why the SSH top (or free) command says I have 32gb of total memory (KiB Mem row) but the cf app command says that I have 7 instances with 2gb each (the configuration I chose).
Am I interpreting one of these incorrectly?
I think you're comparing two different attributes. CF Max allocated memory with max available memory within a CF container. You allocated 2GB per Instance for your app, with 7 instances total. But that doesn't mean they use 2GB memory.
You can view actual memory usage in the IBM Cloud details of an app. Click on the app then click Runtime on the left.
Here is an example screenshot:

CPU usage of Jboss JVM goes upto 99% and stays there

I am doing load testing on my application using jmeter and I have a situation where the cpu usage by the applications jvm goes to 99% and it stays there. Application still work, I am able to login and do some activity. But, it’s understandably slower.
Details of environment:
Server: AMD Optrom, 2.20 Ghz, 8 Core, 64bit, 24 GB RAM. Windows Server 2008 R2 Standard
Application server: jboss-4.0.4.GA
JAVA: jdk1.6.0_25, Java HotSpot(TM) 64-Bit Server VM
JVM settings:
-Xms1G -Xmx10G -XX:MaxNewSize=3G -XX:MaxPermSize=12G -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:+UseCompressedOops -Dsun.rmi.dgc.client.gcInterval=1800000 -Dsun.rmi.dgc.server.gcInterval=1800000
Database: MySql 5.6 (in a different machine)
Jmeter: 2.13
My scenario is that, I make 20 users of my application to log into it and perform normal activity that should not be bringing huge load. Some, minutes into the process, JVM of Jboss goes up and it never comes back. CPU usage will remain like that till JVM is killed.
To help better understand, here are few screen shots.
I found few post which had cup # 100%, but nothing there was same as my situation and could not find a solution.
Any suggestion on what’s to be done will be great.
Regards,
Sreekanth.
To understand the root cause of the high CPU utilization, we need to check the CPU data and thread dumps at same time.
Capture 5-6 thread dumps at the time of the issue. Similarly capture CPU consumption thread-by-thread basis.
Generally the root cause of the CPU issue would be problems with threads like BLOCKED threads, long running threads, dead-lock, long running loops etc. That can be resolved by going through the stacks of the threads.

Meteor crashing

My Meteor app is crashing with the following error:
Unexpected mongo exit code null. Restarting.
=> Exited from signal: SIGKILL
/home/ron/.meteor/packages/meteor-tool/.1.1.3.4sddkj++os.linux.x86_64+web.browser+web.cordova/mt-os.linux.x86_64/dev_bundle/lib/node_modules/fibers/future.js:245
throw(ex);
^
Error: Unable to allocate ArrayBuffer.
This is followed by a call-stack trace.
What is causing this?
Thanks!
This error is probably caused by your operating environment. If its not able to allocate an ArrayBuffer it may be that you don't have enough RAM or some other service is blocking meteor from allocating memory.
This error may occur on the smallest DigitalOcean droplet if that's what you're using.
It's generally recommended you have 1 GB of free ram for Meteor to work properly in development mode.
Something you could use is a swapfile to increase your ram.
Real RAM memory could be replaced with virtual memory but won't be so fast memory... in linux this SO feature is achieved using a swap partition. In windows is using a paging file. Weirdly you can emulate this feature in the linux world using swapspace (or create a traditional swap partition)
sudo apt-get install swapspace
Whatever option you choose will create swap for you and it will help you to start up your meteor app!!!
Just be aware that this will be a more slower experience than real RAM but definitely will work

How do I set the maximum memory size that Redis can use?

To be specific, I only have 1GB of free memory and would like to use only 300MB for Redis. How can I configure it so that it is only uses up to 300MB of memory?
Out of curiosity, what happen when you try to insert a new data and Redis is already used all the memory allocated?
maxmemory is the correct configuration option to prevent Redis from using too much RAM.
If an insert causes maxmemory to be exceeded, the insert operation will sometimes fail.
Redis will do everything in its power to prevent the operation from failing, though. In the newer versions of Redis, you can configure the memory reclaiming policies in the configuration, as well by setting the maxmemory-policy option.
Also, if you have virtual memory options turned on, Redis will begin to store stale data to the disk.
More info:
What does Redis do when it runs out of memory?
You can do that using maxmemory option: maxmemory 314572800 means 300mb.
Since the last answer is from 2011. I am going to write some updated information for users reading in 2019 using Ubuntu 18.04. The configuration file is located in /etc/redis/redis.conf and if you have installed using (default/recommended method) apt install redis-server the default memory limit is set to "0" which practically means there is "no limit" which can be troublesome if user has limited/small amount of RAM/memory. To set your custom memory limit you may simply edit configuration file and type "maxmemory 1gb" as the very first line. Restart redis service for changes to take effect. To verify changes use redis-cli config get maxmemory
Ubuntu 18.04 users may read more here: How to install and configure REDIS on Ubuntu 18.04