SymmetricDS encounters illegalStateException - postgresql

Question

Setup
SymmetricDS version is 3.9.1 (also tried 3.9.0)
Setup is from postgres 9.5.3 to postgres 9.5.3
Windows 10 pc (client node) to windows server 2012 (master node).
The client is my work pc (local) and the windows server is on the other side of the country. The registration url for the client and sync url for the master node are using a dns resolver to get the ip of the server.
The client connects to the server successfully though registration and creates the appropriate triggers on the client. This error occurs while trying to sync.
Getting this message on the client:
PushService - We expected but did not receive an ack for batch liveServer-2.
PushService - This could be because the batch is corrupt. Removing the batch
from staging
The response on the server is:
DataLoaderService - Failed while parsing batch
java.lang.IllegalStateException: Invalid batch data was received: ▼∩┐
∩┐╜TK∩┐╜∩┐╜6►∩┐╜∩┐╜j∩┐╜∩┐╜¶g∩┐╜c∩┐╜☼▲∩┐╜-2x↕
R♀∩┐╜ S∩┐╜∩┐╜J∩┐╜♣xbK∩┐╜$&C~}Z~∩┐╜♀∩┐╜nr∩┐╜!Pr∩┐╜∩┐╜∩┐╜∩┐╜jY∩┐╜Z╚é∩┐╜♣
at org.jumpmind.symmetric.io.stage.SimpleStagingDataWriter.process(SimpleStagingDataWriter.java:194)
at org.jumpmind.symmetric.service.impl.DataLoaderService.loadDataFromTransport(DataLoaderService.java:568)
at org.jumpmind.symmetric.service.impl.DataLoaderService.loadDataFromPush(DataLoaderService.java:398)
at org.jumpmind.symmetric.web.PushUriHandler.push(PushUriHandler.java:83)
at org.jumpmind.symmetric.web.PushUriHandler.handle(PushUriHandler.java:68)
at org.jumpmind.symmetric.web.SymmetricServlet.service(SymmetricServlet.java:114)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:833)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1650)
at org.jumpmind.symmetric.web.HttpMethodFilter.doFilter(HttpMethodFilter.java:62)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1637)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:533)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:190)
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1595)
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:188)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1253)
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:168)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:473)
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1564)
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:166)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1155)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
at org.eclipse.jetty.server.Server.handle(Server.java:561)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:334)
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:279)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:104)
at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:124)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:247)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:140)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131)
at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:243)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:679)
at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:597)
at java.lang.Thread.run(Unknown Source)
How can I resolve this? What could I have done wrong?
I used the same setup between a windows 10 pc (client node) to a Windows 2012 Virtual machine (master node) on the same network and it works fine. This error only seems to occur when I try to connect to a remote server with my pc as the client.
Seeing those odd characters makes me think it was an encoding issue so I tried adding the option to the end of the db.url and also the #db.read.strings.as.bytes=true tag in the engine files
I also tried re-importing the client database from a previous backup encase it was corrupted. Same result on the older version of the database.
If I had to guess perhaps something along the route is encrypting or garbling the packets leading to bad data.

Answer 1

One thing that might be happening is a network device in between your nodes is stripping or modifying HTTP headers. SymmetricDS gzips the data when it goes over the wire and sets the HTTP header Content-Encoding: gzip. If the content-encoding header is not set to gzip when it reaches the other side, you would get junk output like you are seeing.