Spark with neo4j connector, neo4j times out? - scala

Question

EDIT: I have tried several things to get this to work to completion and nothing seems to work:
Updated the neo4j-community.vmoptions:
-server
-Xms4096m
-Xmx4096m
-XX:NewSize=1024m
Also updated my mac to handle more threads.
C02RH2U9G8WM:~ meuser$ ulimit -u
709
C02RH2U9G8WM:~ meuser$ ulimit -a
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
file size (blocks, -f) unlimited
max locked memory (kbytes, -l) unlimited
max memory size (kbytes, -m) unlimited
open files (-n) 256
pipe size (512 bytes, -p) 1
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 709
virtual memory (kbytes, -v) unlimited
Finally ran a test to make sure it wasnt limited harddrive space that was causing the issues. Everything seemed to check out, and during execution I would check my thread usage in another window....
jstack -l 'pid' | grep tid | wc -l
and
ps -elfT | wc -l
Nothing seemed out of hand so I am really confused as to why I am getting the errors below when running spark code in scala connecting to neo4j that runs fine with small set of answers but blows up when I let it rip on everything.
note: in my code I do not use sc.stop() or anything even though I think resources need to be cleared somehow while running a decently sized loop from within spark/scale
I did check the logs, turns out it might be memory related?
2017-10-13 01:17:45.073+0000 ERROR [o.n.b.t.SocketTransportHandler] Fatal error occurred when handling a client connection: unable to create new native thread unable to create new native thread
java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Thread.java:714)
at org.neo4j.kernel.impl.util.Neo4jJobScheduler.schedule(Neo4jJobScheduler.java:94)
at org.neo4j.bolt.v1.runtime.concurrent.ThreadedWorkerFactory.newWorker(ThreadedWorkerFactory.java:68)
at org.neo4j.bolt.v1.runtime.MonitoredWorkerFactory.newWorker(MonitoredWorkerFactory.java:54)
at org.neo4j.bolt.BoltKernelExtension.lambda$newVersions$1(BoltKernelExtension.java:234)
at org.neo4j.bolt.transport.ProtocolChooser.handleVersionHandshakeChunk(ProtocolChooser.java:95)
at org.neo4j.bolt.transport.SocketTransportHandler.chooseProtocolVersion(SocketTransportHandler.java:109)
at org.neo4j.bolt.transport.SocketTransportHandler.channelRead(SocketTransportHandler.java:58)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:349)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:341)
at io.netty.handler.codec.ByteToMessageDecoder.handlerRemoved(ByteToMessageDecoder.java:219)
at io.netty.channel.DefaultChannelPipeline.callHandlerRemoved0(DefaultChannelPipeline.java:631)
at io.netty.channel.DefaultChannelPipeline.remove(DefaultChannelPipeline.java:468)
at io.netty.channel.DefaultChannelPipeline.remove(DefaultChannelPipeline.java:428)
at org.neo4j.bolt.transport.TransportSelectionHandler.switchToSocket(TransportSelectionHandler.java:126)
at org.neo4j.bolt.transport.TransportSelectionHandler.decode(TransportSelectionHandler.java:81)
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:411)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:248)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:349)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:341)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1334)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:349)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:926)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:129)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:642)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:565)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:479)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:441)
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)
at java.lang.Thread.run(Thread.java:745)
I am running these tests on a single mac laptop pro (not clustered or anything yet). An i7 16GB of ram.
I am running some spark scala code that uses the neo4j connector. When I run it on small data sets it works fine, when I let it rip on processing all the data in a neo4j database, it times out after running for a good while (several hours). Usually with me having to reset the community edition restarting the neo4j server. Is there some setting somewhere where I can adjust this time out? Either in Spark/Scala or in neo4j configs?
org.neo4j.driver.v1.exceptions.ClientException: Failed to establish connection with server. Make sure that you have a server with bolt enabled on localhost:7687
at org.neo4j.driver.internal.connector.socket.SocketClient.negotiateProtocol(SocketClient.java:197)
at org.neo4j.driver.internal.connector.socket.SocketClient.start(SocketClient.java:76)
at org.neo4j.driver.internal.connector.socket.SocketConnection.<init>(SocketConnection.java:63)
at org.neo4j.driver.internal.connector.socket.SocketConnector.connect(SocketConnector.java:52)
at org.neo4j.driver.internal.pool.InternalConnectionPool.acquire(InternalConnectionPool.java:113)
at org.neo4j.driver.internal.InternalDriver.session(InternalDriver.java:53)
at org.neo4j.spark.Executor$.execute(Neo4j.scala:360)
at org.neo4j.spark.Neo4jRDD.compute(Neo4j.scala:408)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
at org.apache.spark.scheduler.Task.run(Task.scala:108)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:335)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)