spark-submit netty NoSuchMethodError when using jdbc - scala

Question

I have been trying to run my job using spark-submit and i have a problem when i use jdbc to fetch a DataFrame from a Postgresql.
First of all the jdbc driver is inside my job jar but i had to load the driver like this inside my code
sparkSession.read.option("driver", "org.postgresql.Driver").jdbc(jdbcdn, query, props)
This works fine and the connection to the database is made, i know this because if the server is not found i recieve the appropriate exception from the driver.
But if the connection succeed i always receive the following exception and the job hangs :
17/05/31 10:56:16 ERROR server.TransportRequestHandler: Error sending result StreamResponse{streamId=/jars/bibi-1.0.0-spark.jar, byteCount=3345077, body=FileSegmentManagedBuffer{file=/srv/jobs/bibi-1.0.0-spark.jar, offset=0, length=3345077}} to /127.0.0.1:50087; closing connection
io.netty.handler.codec.EncoderException: java.lang.NoSuchMethodError: io.netty.channel.DefaultFileRegion.<init>(Ljava/io/File;JJ)V
at io.netty.handler.codec.MessageToMessageEncoder.write(MessageToMessageEncoder.java:107)
at io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:658)
at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:716)
at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:651)
at io.netty.handler.timeout.IdleStateHandler.write(IdleStateHandler.java:266)
at io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:658)
at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:716)
at io.netty.channel.AbstractChannelHandlerContext.writeAndFlush(AbstractChannelHandlerContext.java:706)
at io.netty.channel.AbstractChannelHandlerContext.writeAndFlush(AbstractChannelHandlerContext.java:741)
at io.netty.channel.DefaultChannelPipeline.writeAndFlush(DefaultChannelPipeline.java:895)
at io.netty.channel.AbstractChannel.writeAndFlush(AbstractChannel.java:240)
at org.apache.spark.network.server.TransportRequestHandler.respond(TransportRequestHandler.java:194)
at org.apache.spark.network.server.TransportRequestHandler.processStreamRequest(TransportRequestHandler.java:150)
at org.apache.spark.network.server.TransportRequestHandler.handle(TransportRequestHandler.java:111)
at org.apache.spark.network.server.TransportChannelHandler.channelRead(TransportChannelHandler.java:118)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:254)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319)
at org.apache.spark.network.util.TransportFrameDecoder.channelRead(TransportFrameDecoder.java:85)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:787)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:130)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354)
at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.NoSuchMethodError: io.netty.channel.DefaultFileRegion.<init>(Ljava/io/File;JJ)V
at org.apache.spark.network.buffer.FileSegmentManagedBuffer.convertToNetty(FileSegmentManagedBuffer.java:133)
at org.apache.spark.network.protocol.MessageEncoder.encode(MessageEncoder.java:58)
at org.apache.spark.network.protocol.MessageEncoder.encode(MessageEncoder.java:33)
at io.netty.handler.codec.MessageToMessageEncoder.write(MessageToMessageEncoder.java:89)
... 34 more
I tried the following (i am using Gradle)
Exclude netty dependencies from my project
Include a version of netty in my shadowJar
Relocate the included netty
But everything i tried had no effect.
What i am wondering is the problem i had with registering the driver since all the standard ways you can found online does not work with spark/scala/jdbc and i had to use the code above.
It seems to me that the jdbc call is in its own environment and whatever i do in my project gradle has no effect on this environment.
Since that option("driver", "org.postgresql.Driver") was hard to find, I wonder if there is something undocumented here and if i have to find a way to instruct the jdbc runtime on which netty version to use.

Answer 1

Ok so i continued my search and finally found what was going on.
I installed the spark-master and hadoop server myself, since the hadoop jars were going to be on the same server i installed the spark without hadoop.
Hadoop jars were added to spark classpath using the "hadoop classpath" command.
The thing is that hadoop 2.7.3 ship with netty 3.6.2/4.0.23.Final while spark ship with netty 3.8.0/4.0.42.Final
Both were on the classpath in the end provoking the problem.
What i did was to copy both netty jars from spark to all places in hadoop basically upgrading the netty version used by hadoop.
I don't see a problem so far but i use a fraction of what hadoop can do and issues may arise.
EDIT : Another quick fix is to use the spark-with-hadoop tar and NOT add hadoop classpath, that way both use their own jars without conflicting with each other.
This is actually what i ended up doing because i had another jar conflict when accessing sparkUI and it could not be corrected by copying jars like i did with netty.
The conclusion is : NEVER use spark-without-hadoop download.