TCP data sometimes not received by java (or python) server - sockets

Question

I'm developing a system that consists of an arduino mkr1000 that I want to send data via wifi to a java server program running in my local network.
Everything works except the main part: data sent by the arduino is sometimes not received by the server...
I'm using the arduino Wifi101 library to connect to my wifi, get a WiFiClient and send data.
The following code is just a example to demonstrate the problem:
for (int i = 0; i < 3; ++i) {
Serial.println(F("Connecting to wifi"));
const auto status = WiFi.begin("...", "...");
if (status != WL_CONNECTED) {
Serial.print(F("Could not connect to WiFi: "));
switch (status) {
case WL_CONNECT_FAILED:
Serial.println(F("WL_CONNECT_FAILED"));
break;
case WL_DISCONNECTED:
Serial.println(F("WL_DISCONNECTED"));
break;
default:
Serial.print(F("Code "));
Serial.println(status, DEC);
break;
}
} else {
Serial.println(F("WiFi status: WL_CONNECTED"));
WiFiClient client;
if (client.connect("192.168.0.102", 1234)) {
delay(500);
client.print(F("Test "));
client.println(i, DEC);
client.flush();
Serial.println(F("Data written"));
delay(5000);
client.stop();
} else {
Serial.println(F("Could not connect"));
}
WiFi.end();
}
delay(2000);
}
The java server is based on Netty but the same thing with manually creating and reading from a Socket yields the same result.
The testing code is pretty standard with only a simple output (note: in Kotlin):
val bossGroup = NioEventLoopGroup(1)
val workerGroup = NioEventLoopGroup(6)
val serverFuture = ServerBootstrap().run {
group(bossGroup, workerGroup)
channel(NioServerSocketChannel::class.java)
childHandler(object : ChannelInitializer<NioSocketChannel>() {
override fun initChannel(ch: NioSocketChannel) {
ch.pipeline()
.addLast(LineBasedFrameDecoder(Int.MAX_VALUE))
.addLast(StringDecoder())
.addLast(object : ChannelInboundHandlerAdapter() {
override fun channelRead(ctx: ChannelHandlerContext, msg: Any) {
println("msg = $msg")
ctx.close()
}
})
}
})
bind(port).sync()
}
The arduino tells that everything is OK (i.e. writing Data written to the serial console for each iteration) but the server sometimes skips individual messages.
Adding the LoggingHandler from Netty tells me in these cases:
11:28:48.576 [nioEventLoopGroup-3-1] WARN i.n.handler.logging.LoggingHandler - [id: 0x9991c251, L:/192.168.0.20:1234 - R:/192.168.0.105:63845] REGISTERED
11:28:48.577 [nioEventLoopGroup-3-1] WARN i.n.handler.logging.LoggingHandler - [id: 0x9991c251, L:/192.168.0.20:1234 - R:/192.168.0.105:63845] ACTIVE
In the cases where the message is received it tells me:
11:30:01.392 [nioEventLoopGroup-3-6] WARN i.n.handler.logging.LoggingHandler - [id: 0xd51b7bc3, L:/192.168.0.20:1234 - R:/192.168.0.105:59927] REGISTERED
11:30:01.394 [nioEventLoopGroup-3-6] WARN i.n.handler.logging.LoggingHandler - [id: 0xd51b7bc3, L:/192.168.0.20:1234 - R:/192.168.0.105:59927] ACTIVE
11:30:01.439 [nioEventLoopGroup-3-6] WARN i.n.handler.logging.LoggingHandler - [id: 0xd51b7bc3, L:/192.168.0.20:1234 - R:/192.168.0.105:59927] READ: 8B
+-------------------------------------------------+
| 0 1 2 3 4 5 6 7 8 9 a b c d e f |
+--------+-------------------------------------------------+----------------+
|00000000| 54 65 73 74 20 32 0d 0a |Test 2.. |
+--------+-------------------------------------------------+----------------+
11:30:01.449 [nioEventLoopGroup-3-6] WARN i.n.handler.logging.LoggingHandler - [id: 0xd51b7bc3, L:/192.168.0.20:1234 - R:/192.168.0.105:59927] CLOSE
11:30:01.451 [nioEventLoopGroup-3-6] WARN i.n.handler.logging.LoggingHandler - [id: 0xd51b7bc3, L:/192.168.0.20:1234 ! R:/192.168.0.105:59927] READ COMPLETE
11:30:01.453 [nioEventLoopGroup-3-6] WARN i.n.handler.logging.LoggingHandler - [id: 0xd51b7bc3, L:/192.168.0.20:1234 ! R:/192.168.0.105:59927] INACTIVE
11:30:01.464 [nioEventLoopGroup-3-6] WARN i.n.handler.logging.LoggingHandler - [id: 0xd51b7bc3, L:/192.168.0.20:1234 ! R:/192.168.0.105:59927] UNREGISTERED
With my understanding this means that the TCP packets are indeed received but in the faulty cases the IO thread from Netty is waiting to read the TCP data but does never continue...
The same problem exists when trying with a rudimentary python server (just waiting for a connection and printing the received data).
I confirmed the data is sent by using tcpflow on Arch Linux with the arguments -i any -C -g port 1234.
I even tried the server on a Windows 7 machine with the same results (TCP packets confirmed with SmartSniff).
Strangely using a java server to send the data always and reproducibly is received...
Does anybody have any idea to solve the problem or at least how to diagnose?
PS: Maybe it is important to note that with tcpflow (i.e. on linux) I could watch the TCP packets being resent to the server.
Does this mean the server is receiving the packets but not sending an ACK?
SmartSniff didn't show the same behavior (but maybe I used wrong options to display the resent packets).

Answer 1

In the meantime I send messages to acknowledge receiving another message. If the acknowledgement is not received the message is sent again.
For anyone with the same problem:
While testing something different I updated the wifi firmware of the board to the latest version 19.5.2. Since then I haven't noticed any lost data. Maybe this was the problem.
See Check WiFi101 Firmware Version and Firmware and certificates Updater.
Note: I couldn't get the sketches to run with the Arduino IDE but with PlatformIO.