despite partitioning I keep blowing up the spark cluster - scala

Question

I have a spark job that blows up our CDH cluster in one of two ways depending on how I partition things. The purpose of this job is to generate anywhere between 1 and 210,094,780,875 sets of four integers. The job is being submitted via spark-submit, master is set to YARN. Below is the code snip germane to this question:
// build rdd and let cluster build up the ngram list
val streamList_rdd = sc.parallelize(streamList).repartition(partitionCount)
val rdd_results = streamList_rdd.flatMap { x => x.toList }
println(rdd_results.count())
streamList is a list of generators that have been seeded with a floor/ceiling value (a tuple containing two Ints) that will generate sets of four integers bounded by floor/ceiling. The idea is to farm out the generation work across the cluster and that's where the front falls off. If partitionCount is too low (and thus the size of each partition is large), the workers blow up due to lack of memory. If partitionCount is high (and thus the size of each partition is manageable from a memory perspective), you start seeing errors like this one:
java.io.IOException: Connection reset by peer
at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
at sun.nio.ch.IOUtil.read(IOUtil.java:192)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:379)
at io.netty.buffer.PooledUnsafeDirectByteBuf.setBytes(PooledUnsafeDirectByteBuf.java:313)
at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:881)
at io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:242)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:119)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354)
at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
at java.lang.Thread.run(Thread.java:745)
The memory issue I understand - what I'm not understanding is why there are issues with having a high partition count (~100k or more). Is there a way for me to make this work while preserving YARN's role in managing cluster resources?

Answer 1

Given the amount of data, and the presence of memory errors, I think you need to assign more cluster resources.
Increasing partitions improves parallelism but at the cost of consuming more resources on an already insufficiently sized cluster. I also suspect the repartition operation causes a shuffle which is an expensive operation at the best of times, very bad (catastrophic!) when you have enough data to out-of-memory. But without logs, that is conjecture.
The cause of heartbeat failure is likely either an executor is under such heavy load it fails to respond in time, or the process has crashed / been killed by YARN...