Experiencing "Connection reset by peer" while using grpc on Mesosphere Marathon - haproxy

Question

I'm getting tons of these errors deploying a grpc (1.5.0) java app on marathon.
Aug 01, 2017 4:49:12 PM io.grpc.netty.NettyServerTransport notifyTerminated
SEVERE: Transport failed
java.io.IOException: Connection reset by peer
at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
at sun.nio.ch.IOUtil.read(IOUtil.java:192)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380)
at io.netty.buffer.PooledUnsafeDirectByteBuf.setBytes(PooledUnsafeDirectByteBuf.java:288)
at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:1100)
at io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:372)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:123)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:644)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:579)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:496)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:458)
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)
at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138)
at java.lang.Thread.run(Thread.java:745)
EDIT:
I'm also noticing
[pool-1-thread-4] --- 2017-08-02 12:31:09,465 WARN e.a.s.grcp.ConnectionStatusServer - requests.size 1
[nioEventLoopGroup-2-1] --- 2017-08-02 12:31:09,465 DEBUG i.n.h.c.http2.Http2ConnectionHandler - [id: 0x0a5278e3, L:0.0.0.0/0.0.0.0:31169 ! R:/10.124.56.175:47384] Sending GOAWAY failed: lastStreamId '7', errorCode '2', debugData 'Connection reset by peer'. Forcing shutdown of the connection.java.io.IOException: Broken pipe
---> at sun.nio.ch.FileDispatcherImpl.writev0(Native Method)
---> at sun.nio.ch.SocketDispatcher.writev(SocketDispatcher.java:51)
---> at sun.nio.ch.IOUtil.write(IOUtil.java:148)
---> at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:504)
---> at io.netty.channel.socket.nio.NioSocketChannel.doWrite(NioSocketChannel.java:432)
---> at io.netty.channel.AbstractChannel$AbstractUnsafe.flush0(AbstractChannel.java:856)
---> at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.flush0(AbstractNioChannel.java:362)
---> at io.netty.channel.AbstractChannel$AbstractUnsafe.flush(AbstractChannel.java:823)
---> at io.netty.channel.DefaultChannelPipeline$HeadContext.flush(DefaultChannelPipeline.java:1296)
---> at io.netty.channel.AbstractChannelHandlerContext.invokeFlush0(AbstractChannelHandlerContext.java:776)
---> at io.netty.channel.AbstractChannelHandlerContext.invokeFlush(AbstractChannelHandlerContext.java:768)
---> at io.netty.channel.AbstractChannelHandlerContext.flush(AbstractChannelHandlerContext.java:749)
---> at io.netty.handler.codec.http2.Http2ConnectionHandler.onError(Http2ConnectionHandler.java:573)
---> at io.grpc.netty.AbstractNettyHandler.exceptionCaught(AbstractNettyHandler.java:84)
---> at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:285)
---> at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:264)
---> at io.netty.channel.AbstractChannelHandlerContext.fireExceptionCaught(AbstractChannelHandlerContext.java:256)
---> at io.netty.channel.DefaultChannelPipeline$HeadContext.exceptionCaught(DefaultChannelPipeline.java:1301)
---> at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:285)
---> at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:264)
---> at io.netty.channel.DefaultChannelPipeline.fireExceptionCaught(DefaultChannelPipeline.java:914)
---> at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.handleReadException(AbstractNioByteChannel.java:104)
---> at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:145)
---> at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:644)
---> at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:579)
---> at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:496)
---> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:458)
---> at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)
---> at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138)
---> at java.lang.Thread.run(Thread.java:745)
--->