Golang raw sockets losing packets? - sockets

Question

I'm writing a simple program in Golang to capture TCP/IP packets using raw sockets:
package main
import (
"fmt"
"log"
"net"
)
func main() {
netaddr, err := net.ResolveIPAddr("ip", "127.0.0.1")
if err != nil {
log.Fatal(err)
}
conn, err := net.ListenIP("ip:tcp", netaddr)
if err != nil {
log.Fatal(err)
}
packet := make([]byte, 64*1024)
numPackets := 0
totalLen := 0
for {
packetLen, _, err := conn.ReadFrom(packet)
if err != nil {
log.Fatal(err)
}
dataOffsetWords := (packet[12] & 0xF0) >> 4
dataOffset := 4 * dataOffsetWords
payload := packet[dataOffset:packetLen]
numPackets += 1
totalLen += len(payload)
fmt.Println("Num packets:", numPackets, ", Total len:", totalLen)
}
}
When I compare the number of packets which the program receives and the total amount of data they contain to the number of packets Wiresharks sees and the total data transmitted, I know I've lost 15-30 % of all packets and data on every run.
Why?
The only thing that comes to my mind is that the application is not fast enough to receive the packets, but that's odd. (I'm communicating on localhost and sending ~ 17 MB of data.) Goreplay however uses something similar and works.
The traffic I'm receiving is created by curl-ing onto a locally running Python server (http.server) and sending a huge file in the request body. The Python server downloads the whole body successfully.

Answer 1

My initial suspiction was right: by modifying the receiving Python server so that it does not read all incoming data at once:
post_body = self.rfile.read(content_len)
but rather in a loop with a 10 ms delay between iterations:
while (total_len < content_len):
post_body = self.rfile.read(min(16536, content_len - total_len))
total_len += len(post_body)
time.sleep(1 / 100.)
I receive all data in the Golang and the number of packets processed is approximately te same (I assume there will only rarely be a match as the packets received in Golang were already processed, i.e. put back into order. Or at least I hope, because there's no much documentation.)
Also, the packet loss changes when the delay is adjusted.