Kubernetes services communication by REST: Stream ended unexpectedly - rest

Question

There are two services deployed into Kubernetes cluster. Service_1 exposes REST API, and one part of it is the method for file content uploading, so POST request with "Content-Type: multipart/form-data" is used.
The sample of real request, which is sending from Service_2 is:
Request DefaultFullHttpRequest(decodeResult: success, version: HTTP/1.1,
content: UnpooledHeapByteBuf(freed))
POST /engine-rest/deployment/create HTTP/1.1
Accept: application/json
User-Agent: process
Content-Type: multipart/form-data; boundary=28319d96a8c54b529aa9159ad75edef9
Content-Length: 4028
Host: service.cluster.ip:8080
The request cannot be processed and failed with an exception:
30-Mar-2017 18:17:29.623 WARNING [http-nio-8080-exec-2] org.camunda.bpm.engine.rest.exception.RestExceptionHandler.toResponse org.camunda.bpm.engine.rest.exception.RestException: multipart/form-data cannot be processed
at org.camunda.bpm.engine.rest.mapper.MultipartPayloadProvider.parseRequest(MultipartPayloadProvider.java:93)
at org.camunda.bpm.engine.rest.mapper.MultipartPayloadProvider.readFrom(MultipartPayloadProvider.java:71)
at org.camunda.bpm.engine.rest.mapper.MultipartPayloadProvider.readFrom(MultipartPayloadProvider.java:49)
at org.jboss.resteasy.core.interception.MessageBodyReaderContextImpl.proceed(MessageBodyReaderContextImpl.java:105)
at org.jboss.resteasy.plugins.interceptors.encoding.GZIPDecodingInterceptor.read(GZIPDecodingInterceptor.java:63)
at org.jboss.resteasy.core.interception.MessageBodyReaderContextImpl.proceed(MessageBodyReaderContextImpl.java:108)
at org.jboss.resteasy.core.MessageBodyParameterInjector.inject(MessageBodyParameterInjector.java:169)
at org.jboss.resteasy.core.MethodInjectorImpl.injectArguments(MethodInjectorImpl.java:136)
at org.jboss.resteasy.core.MethodInjectorImpl.invoke(MethodInjectorImpl.java:159)
at org.jboss.resteasy.core.ResourceMethod.invokeOnTarget(ResourceMethod.java:257)
at org.jboss.resteasy.core.ResourceMethod.invoke(ResourceMethod.java:222)
at org.jboss.resteasy.core.ResourceLocator.invokeOnTargetObject(ResourceLocator.java:159)
at org.jboss.resteasy.core.ResourceLocator.invoke(ResourceLocator.java:92)
at org.jboss.resteasy.core.SynchronousDispatcher.getResponse(SynchronousDispatcher.java:542)
at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:524)
at org.jboss.resteasy.core.SynchronousDispatcher.invokePropagateNotFound(SynchronousDispatcher.java:169)
at org.jboss.resteasy.plugins.server.servlet.ServletContainerDispatcher.service(ServletContainerDispatcher.java:212)
at org.jboss.resteasy.plugins.server.servlet.FilterDispatcher.doFilter(FilterDispatcher.java:59)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:239)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at org.camunda.bpm.engine.rest.filter.CacheControlFilter.doFilter(CacheControlFilter.java:41)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:239)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:219)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:106)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:502)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:142)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:79)
at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:617)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:88)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:518)
at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1091)
at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:668)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1527)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1484)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.commons.fileupload.MultipartStream$MalformedStreamException: Stream ended unexpectedly
at org.apache.commons.fileupload.MultipartStream.readHeaders(MultipartStream.java:538)
at org.apache.commons.fileupload.FileUploadBase$FileItemIteratorImpl.findNextItem(FileUploadBase.java:999)
at org.apache.commons.fileupload.FileUploadBase$FileItemIteratorImpl.<init>(FileUploadBase.java:965)
at org.apache.commons.fileupload.FileUploadBase.getItemIterator(FileUploadBase.java:331)
at org.camunda.bpm.engine.rest.mapper.MultipartPayloadProvider.parseRequest(MultipartPayloadProvider.java:87)
... 38 more
What might be the reason of this error? I understand that the question doesn't have the direct answer, but I hope someone is able point me into the right way for additional investigation.
P.S. GET type requests of this API work fine.

Answer 1

Sorted out! If someone is interested or acccidentaly faced the exatly same problem, the root cause was in CLRF, which are required for multi-part requests, when we were using System.lineSeparator() and thus it hadn't worked under Linux. Yeah, so simple.