Not able to set right memory options in Spark - scala

Question

I am training a LDA model on wikipedia articles(4 million docs, ~14GB data). I am running a scala script on one machine with ~98GB memory. I run the scala code in spark shell with following params:
$SPARK_HOME/bin/spark-shell --executor-memory 2G --driver-memory 25G --total-executor-cores 10 --conf spark.driver.maxResultSize=50g
Code snippet:
val lda = new LDA().setOptimizer(new OnlineLDAOptimizer()).setK(numTopics).setMaxIterations(maxIterations)
val ldaModel = lda.run(lda_countVector)
I get the following error when I execute lda.run():
scala> val ldaModel = lda.run(lda_countVector)
16/06/30 12:53:45 WARN BLAS: Failed to load implementation from: com.github.fommil.netlib.NativeSystemBLAS
16/06/30 12:53:45 WARN BLAS: Failed to load implementation from: com.github.fommil.netlib.NativeRefBLAS
[Stage 21:==============================> (238 + 85) / 408]16/06/30 13:35:59 ERROR Executor: Exception in task 315.0 in stage 21.0 (TID 2803)
java.lang.OutOfMemoryError: Java heap space
at java.util.Arrays.copyOf(Arrays.java:3236)
at java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:118)
at java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:93)
at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:153)
at java.io.ObjectOutputStream$BlockDataOutputStream.write(ObjectOutputStream.java:1853)
at java.io.ObjectOutputStream.write(ObjectOutputStream.java:709)
at org.apache.spark.util.Utils$.writeByteBuffer(Utils.scala:183)
at org.apache.spark.scheduler.DirectTaskResult$$anonfun$writeExternal$1.apply$mcV$sp(TaskResult.scala:52)
at org.apache.spark.util.Utils$.tryOrIOException(Utils.scala:1204)
at org.apache.spark.scheduler.DirectTaskResult.writeExternal(TaskResult.scala:49)
at java.io.ObjectOutputStream.writeExternalData(ObjectOutputStream.java:1459)
at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1430)
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:348)
at org.apache.spark.serializer.JavaSerializationStream.writeObject(JavaSerializer.scala:44)
at org.apache.spark.serializer.JavaSerializerInstance.serialize(JavaSerializer.scala:101)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:256)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
I have tried various settings for memory like increasing driver memory, increasing maxResultSize memory, executor memory etc, but I still get the same error. If I reduce maxResultSize below 40GB, then I get maxResultSize error.
Please help me figure out the right memory settings. What should be done for a typical application like this?
Thanks.

Answer 1

In spark-shell you didn't provide a master url. This means that you run in local mode. So executor memory setting will be ignored. The number of cores will default the number of CPU-s. To be most conservative run with -master local[1]. This means only 1 core
You could increase the driver memory to 80G , also set spark.memory.storageFraction to 0.1. This takes memory used to cache Rdd's and gives more to the reducers.