I tried to do an algorithm(Fast Fourier Transform, FFT) in Spark.
When data is over 64m(Though it is really small.), The console shows the message:
java.lang.OutOfMemoryError: java heap space.
I use 32-bit Ubuntu system and Spark-1.6.0. And I use the Scala IDE, eclipse.
I use local mode and standalone mode. When the data is smaller than 32m, it works well, but when the data is larger than 64m, It cannot work.
I have tried to set the JVM settings to -Xms1000m and -Xmx4000m. I also tried to add .set("spark.exetutor.memory", "8g") when creating the SparkConf in the code. But both of them do not work.
How can I fix this?
if you work with spark.master = local, then the relevant value to adjust is spark.driver.memory. Note that this option needs to be set before the JVM (i.e. the driver) is launched in local mode, so modifiyng the existing SparkContext won't help, because the JVM is already launched.
Related
I'm getting Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
when running some main object in scala with IntelliJ Idea,
with the "^R" shortcut or Right click-> run 'mainObjName'
This program runs without problem if using sbt:
$ JAVA_OPTS="-Xmx4096m" sbt
sbt:prj> run mainobj
With the search results got from this site, I tried to set the heap space to 4096m in :
1.Preferences->Build Tools->sbt->Maximum heap size,MB
2.Preferences->Compiler->Build process heap size(Mbytes)
3.Preferences->Compiler->Scala Compiler->Scala Compiler Server->JVM Maximum heap size,MB
4.Preferences->Build Tools->Maven->Runner->VM Options (with -Xmx4096m)
But none of these works.
I'm running a memory hungry program, and this is clearly not a build time error but some thing happened during runtime.
So, How to change the maximum heap size for running a program in IntelliJ Idea?
btw, the program runs fine on linux workstation with much more memory, but seems the default heap size configuration is the same on the IntelliJ Idea on MacOS vs Linux. seems java consumes more memory on MacOS?
It seems, that all the options you've tried are setting the memory for various compiler facilities.
You could try to change the memory available for the runtime by setting the VM options to -Xmx4096m in the configuration you are running. You can find the configuration with your mainObjName under Run -> Edit Configurations... -> Application
When I run the program from PyDev/Eclipse, it runs out of memory and gives the following error:
java.lang.OutOfMemoryError: java.lang.OutOfMemoryError: GC overhead
limit exceeded
It goes away when I pass the -J-Xmx2048 limit from the command line. So clearly, this can be solved if PyDev can read these somehow.
In Eclipse, I tried setting this values in the run options (in eclipse) as program & vm arugments, but I get the same error. I also tried setting the JAVA_MEM option, but that doesn't help either.
Any ideas how I can instruct PyDev/Eclipse to read these arguments?
It turns out that the Run As -> Run Configurations -> Arguments -> Vm Arguments for Jython also expect Java arguments and not Jython ones :/
Setting the VM arguments to -Xmx2048 fixed the issue. The memory usage for java now peaks to 2G (earlier 1G) confirming the fix.
My Javafx application was taking so much memory.
I read about -Xmx VM arguments in Eclipse and applied.
It gave me good result. After using this maximum memory was 76000k. But when I created build of that project it again start consuming more than 10000k and contentiously increasing.
Suggest some ideas and edit if i'm wrong in my language.
I'm running GWT from within IntelliJ. (Plain GWT, IntelliJ v9). I'm able to run my application via my "GWT development mode" configuration, but when I try to run it through my "local tomcat" configuration, I get a bunch of incomprehensible error messages referring to Oracle and all sorts of weird stuff that I don't use, followed by this error which is sorted to the bottom of all the others:
Error: Out of memory; to increase the amount of memory, use the -Xmx flag at startup (java -Xmx128M ...)
I'm guessing that this error is the root cause.
According to my understanding, there is a GWT compilation step which runs in a JVM separate to both IntelliJ and tomcat, and so I'm unsure where to set the -Xmx parameter.
My question is: where do I find this -Xmx parameter? (And: am I on the right track, taking this error message at face value, or is it a symptom of deeper problems?)
You can configure the heap in the GWT facet settings:
As CrazyCoder just said you can increase the amount of memory for gwt compiler in the GWT Facet settings. The GWT Compiler running out of memory on bigger apps is very common if you don`t increase the amount of memory. Most projects should be fine with 512m, but on large projects I already needed more memory.
So this is not some out of memory error caused by some bad design on your behalf.
In my application, I set it to 1024 but it gives me out of memory after 7-9 page refresh on dev mode. And after that I tried it to optimize by doing performance hack, I separated my *.gwt.xml for every gwt module for specific browsers by setting:
MySampleModule_FF.gwt.xml
<set-property name="user.agent" value="gecko1_8"/>
And I run my dev mode with this gwt.xml for Firefox. In addition to it also separated internationalization as well.
After that my out of memory issues considerably decreased.
32 bit JDK also sometimes causes Out Of Memory error. Try using 64 bit JDK.
To those who get the same error under Eclipse, try to restore cache files (if they were removed accidentally in WAR folder ), this way I solve the same problem under my Eclipse project.
I cannot get JBoss Portal to start from Eclipse, though the AS alone starts fine, and the Portal starts correctly as well, when started from the command line as opposed to from within Eclipse. I'm running in Windows, with 3GB. Suggestions? Thank you.
I've spend hours to discover this, and almost gave up and started to use JBoss out of Eclipse.
In order to increase your JBoss vmargs when starting it from Eclipse you have to change JBoss launch configuration. If you change standalone.conf, nothing happens because Eclipse doesn't use it.
So, to change JBoss vmargs in Eclipse, you have to go to "Servers" tab, right click on your Jboss instance, and select "Open".
It will appear a new window. In the first section, you have a option: "Open launch configuration". When you click there, you'll see the textbox to change vmargs.
Hope this helps you!
There are different types of OutOfMemory errors:
java.lang.OutOfMemoryError: Java heap space
Increase the -Xms and -Xmx. I'd make sure they are set at least 256m and generally it's a good idea to set them to the same value.
java.lang.OutOfMemoryError: PermGen space
Add either -XX:+CMSPermGenSweepingEnabled or increase the PermGen size: -XX:PermSize=256m
java.lang.OutOfMemoryError: GC overhead limit exceeded
Add more heap, the garbage collector can't free enough memory with each cycle. Also try turning on GC logging.
java.lang.OutOfMemoryError: unable to create new native thread
Decrease your heap :) This means that you have too much memory allocated to the heap that the OS doesn't have enough memory to create threads..
Two last things, the above can be configured in jboss/bin/run.conf.
Also when starting JBoss see what -X parameters are being passed to the JVM, it prints this information by default, verify that it's what you expect it to be.
You need to increase the memory you're allocating to Java, in particular heap space and PermGen. This article is highly relevant. It mentions that this issue often occurs with Eclipse and JBoss (since both are fairly large), and provides a solution (adjusting the command-line flags).
What are you using for running portal from eclipse? Maybe Jboss tools can help you
http://www.jboss.org/tools
According to my experiments, all options of vmargs set in eclipse.ini, plays only once - when creating a new workspace. When you want to change the options in the existing workspace, use run/debug configuration as in https://stackoverflow.com/a/10814631/715269. vmargs in ini won't be read any more.
Be careful, you should set -XX:MaxPermSize=...M, not -XX:PermSize=..., the last sets minimal, starting PermSize.
ad. Jeremy. It is senseless to put mins and maxs to the same value. You deprive Eclipse of adaptability. -Xms and -Xmx ( heap) and PermGen and MaxPermGen should be different. (MaxPermGen =256 by default)