I feel that there may be a problem when the receiving server returns. Relevant code follows:
const RecvBufferSize = 1024 * 64
for {
readBuff := make([]byte, RecvBufferSize)
n, err := udp_client.conn.Read(readBuff)
if err != nil {
log.Error("socket recv error.", zap.String("errMsg", err.Error()))
return
}
select {
case udp_client.recvMessage <- readBuff[:n]:
case <-udp_client.closeTest:
log.Info("close recvFrom", zap.String("Stack", string(debug.Stack())))
close(udp_client.recvMessage)
return
}
}
Each time this code receives the information returned by the server, it will re apply for an address of []byte. Will this put pressure on the GC when the received data volume is compared? Is there any good solution?
I'm using Go for performance testing.
Related
I've accidentally spotted a bug when parts of a message from previous connection go to the next message.
I have a basic server with client. I have removed all the error handling to avoid bloating the examples too much.
Also I've replaced some Printf's with time.Sleep since I just don't have a chance to break the connection in time to reproduce the bug because it reads the data too fast.
The "package" is a simple structure, where the first 4 bytes is the length and then goes the content.
Client code:
package main
import (
"encoding/binary"
"fmt"
"net"
)
func main() {
conn, _ := net.Dial("tcp", "0.0.0.0:8081")
defer conn.Close()
str := "msadsakdjsajdklsajdklsajdk"
// Creating a package
buf := make([]byte, len(str)+4)
copy(buf[4:], str)
binary.LittleEndian.PutUint32(buf[:4], uint32(len(str)))
for {
_, err := conn.Write(buf)
if err != nil {
fmt.Println(err)
return
}
}
}
Server code:
package main
import (
"encoding/binary"
"fmt"
"net"
"sync"
"time"
)
func ReadConnection(conn net.Conn, buf []byte) (err error) {
maxLen := cap(buf)
readSize := 0
for readSize < maxLen {
// instead of Printf
time.Sleep(time.Nanosecond * 10)
readN, err := conn.Read(buf[readSize:])
if err != nil {
return err
}
readSize += readN
}
return nil
}
func handleConnection(conn net.Conn, waitGroup *sync.WaitGroup) {
waitGroup.Add(1)
defer conn.Close()
defer waitGroup.Done()
fmt.Printf("Serving %s\n", conn.RemoteAddr().String())
var packageSize int32 = 0
int32Buf := make([]byte, 4)
for {
// read the length
conn.Read(int32Buf)
packageSize = int32(binary.LittleEndian.Uint32(int32Buf))
// assuming the length should be 26
if packageSize > 26 {
fmt.Println("Package size error")
return
}
// read the content
packageBuf := make([]byte, packageSize)
if err := ReadConnection(conn, packageBuf); err != nil {
fmt.Printf("ERR: %s\n", err)
return
}
// instead of Printf
time.Sleep(time.Nanosecond * 100)
}
}
func main() {
//establish connection
listener, _ := net.Listen("tcp", "0.0.0.0:8081")
defer listener.Close()
waitGroup := sync.WaitGroup{}
for {
conn, err := listener.Accept()
if err != nil {
break
}
go handleConnection(conn, &waitGroup)
}
waitGroup.Wait()
}
So for some reason, int32Buf receives the last 2 bytes from a previous message (d, k) and the first 2 bytes of the length, resulting in [107, 100, 26, 0] bytes slice, when it should be [26, 0, 0, 0].
And of course, the rest of the data contains remaining two zeroes:
conn.Read(int32Buf)
You need to check the return value of conn.Read and compare it against your expectations. You are assuming in your code that conn.Read will always completely fill the given buffer of 4 bytes.
This assumption is wrong, i.e. it might actually read less data. Specifically it might read only 2 bytes in which case you'll end up with \x1a\x00\x00\x00 in your buffer which still translates to a message length of 26. Only, the first 2 bytes of the message will actually be the last 2 bytes of the length which were not included in the last read. This means after reading the 26 bytes it will not have read the full message. 2 bytes are legt and will be included into the next message - this is what you observed.
To be sure that the exact size of the buffer is read check the return values of conn.Read or use io.ReadFull. After you've done this it works as expected (from the comment):
Ok, now it works perfect
So why does this happened only in context of a new connection? Maybe because the additional load due to another connection changed the behavior slightly but significantly enough. Still, these are not the data read from a different connection but data from the current one contrary to the description in the question. This could be easily checked by using different messages with different clients.
I'm looking for the cleanest way in Golang to transfer a message through (i.e. act as an SMTP proxy) while performing some manipulation on the message body html (e.g. adding an open tracking pixel - not yet coded).
The net/mail package includes a method ReadMessage that parses mail headers into a map, and gives you an io.Reader for the body. This is necessary to determine the MIME parts of the body for processing, rather than just io.Copying them through. (the simple stub version of this function, shown in the block comment, does just that).
The following function copies an incoming mail "src" to an outgoing mail stream "dest". (The calling code sets these up as DotReader and DotWriter which takes care of most of the "dot" processing needed for RFC5321.
// Processing of email body via IO stream functions
package main
import (
"bufio"
"io"
"log"
"net/mail"
"strings"
)
/* If you just want to pass through the entire mail headers and body, you can just use
the following alernative:
func MailCopy(dst io.Writer, src io.Reader) (int64, error) {
return io.Copy(dst, src)
}
*/
// MailCopy transfers the mail body from downstream (client) to upstream (server)
// The writer will be closed by the parent function, no need to close it here.
func MailCopy(dst io.Writer, src io.Reader) (int64, error) {
var totalWritten int64
const smtpCRLF = "\r\n"
message, err := mail.ReadMessage(bufio.NewReader(src))
if err != nil {
return totalWritten, err
}
// Pass through headers. The m.Header map does not preserve order, but that should not matter.
for hdrType, hdrList := range message.Header {
for _, hdrVal := range hdrList {
hdrLine := hdrType + ": " + hdrVal + smtpCRLF
log.Print("\t", hdrLine)
bytesWritten, err := dst.Write([]byte(hdrLine))
totalWritten += int64(bytesWritten)
if err != nil {
return totalWritten, err
}
}
}
// Blank line denotes end of headers
bytesWritten, err := io.Copy(dst, strings.NewReader(smtpCRLF))
totalWritten += int64(bytesWritten)
if err != nil {
return totalWritten, err
}
// Copy the body
bytesWritten, err = io.Copy(dst, message.Body)
totalWritten += int64(bytesWritten)
if err != nil {
return totalWritten, err
}
return totalWritten, err
}
It does seem necessary to build this, because there is no net/mail.WriteMessage() method.
the header order is always randomised by Golang's map functionality. This seems harmless in my tests
A forced CRLF needs to be put in between the end of the headers and the body, as per RFCs. DotWriter takes care of the terminating dot.
The function shown above works, I was wondering if there is a better way to do this?
Today when I try to send 100M data to my server (a very simple TCP server also written in Golang), I found that the TCPConn.Write method returns 104857600 and nil error and then I close the socket. But my server only receives very little data. I think it is because Write method works in async mode, so although the method returns 104857600, only a little data is sent to the server. So I want to know whether there is a way to set the Write work in sync mode or how to detect whether all data is sent to the server from the socket.
The code is as follows:
server:
const ListenAddress = "192.168.0.128:8888"
func main() {
var l net.Listener
var err error
l, err = net.Listen("tcp", ListenAddress)
if err != nil {
fmt.Println("Error listening:", err)
os.Exit(1)
}
defer l.Close()
fmt.Println("listen on " + ListenAddress)
for {
conn, err := l.Accept()
if err != nil {
fmt.Println("Error accepting: ", err)
os.Exit(1)
}
//logs an incoming message
fmt.Printf("Received message %s -> %s \n", conn.RemoteAddr(), conn.LocalAddr())
// Handle connections in a new goroutine.
go handleRequest(conn)
}
}
func handleRequest(conn net.Conn) {
defer conn.Close()
rcvLen := 0
rcvData := make([]byte,20 * 1024 * 1024) // 20M
for {
l , err := conn.Read(rcvData)
if err != nil {
fmt.Printf("%v", err)
return
}
rcvLen += l
fmt.Printf("recv: %d\r\n", rcvLen)
conn.Write(rcvData[:l])
}
}
Client:
conn, err := net.Dial("tcp", "192.168.0.128:8888")
if err != nil {
fmt.Println(err)
os.Exit(-1)
}
defer conn.Close()
data := make([]byte, 500 * 1024 * 1024)
length, err := conn.Write(data)
fmt.Println("send len: ", length)
The output of the client:
send len: 524288000
The output of the server:
listen on 192.168.0.128:8888
Received message 192.168.0.2:50561 -> 192.168.0.128:8888
recv: 166440
recv: 265720
EOF
I know if I can make the client wait for a while by SetLinger method, the data will be all sent to the server before the socket is closed. But I want to find a way to make the socket send all data before returns without calling SetLinger().
Please excuse my poor English.
Did you poll the socket before trying to write?
Behind the socket is your operating system's tcp stack. When writing on a socket, you push bytes to the send buffer. Your operating system then self determines when and how to send. If the receiving end has no buffer space.available in their receice buffer, your sending end knows this and will not put any more information in the send buffer.
Make sure your send buffer has enough space for whatever you are trying to send next. This is done by polling the socket. This method is usually called Socket.Poll. I.recommend you ccheck the golang docs for the exact usage.
You are not handling the error returned by conn.Read correctly. From the docs (emphasis mine):
When Read encounters an error or end-of-file condition after successfully reading n > 0 bytes, it returns the number of bytes read. It may return the (non-nil) error from the same call or return the error (and n == 0) from a subsequent call. [...]
Callers should always process the n > 0 bytes returned before considering the error err. Doing so correctly handles I/O errors that happen after reading some bytes and also both of the allowed EOF behaviors.
Note that you are re-inventing io.Copy (albeit with an excessive buffer size). Your server code can be rewritten as:
func handleRequest(conn net.Conn) {
defer conn.Close()
n, err := io.Copy(conn, conn)
}
Talk is cheap, so here we go the simple code:
package main
import (
"fmt"
"time"
"net"
)
func main() {
addr := "127.0.0.1:8999"
// Server
go func() {
tcpaddr, err := net.ResolveTCPAddr("tcp4", addr)
if err != nil {
panic(err)
}
listen, err := net.ListenTCP("tcp", tcpaddr)
if err != nil {
panic(err)
}
for {
if conn, err := listen.Accept(); err != nil {
panic(err)
} else if conn != nil {
go func(conn net.Conn) {
buffer := make([]byte, 1024)
n, err := conn.Read(buffer)
if err != nil {
fmt.Println(err)
} else {
fmt.Println(">", string(buffer[0 : n]))
}
conn.Close()
}(conn)
}
}
}()
time.Sleep(time.Second)
// Client
if conn, err := net.Dial("tcp", addr); err == nil {
for i := 0; i < 2; i++ {
_, err := conn.Write([]byte("hello"))
if err != nil {
fmt.Println(err)
conn.Close()
break
} else {
fmt.Println("ok")
}
// sleep 10 seconds and re-send
time.Sleep(10*time.Second)
}
} else {
panic(err)
}
}
Ouput:
> hello
ok
ok
The Client writes to the Server twice. After the first read, the Server closes the connection immediately, but the Client sleeps 10 seconds and then re-writes to the Server with the same already closed connection object(conn).
Why can the second write succeed (returned error is nil)?
Can anyone help?
PS:
In order to check if the buffering feature of the system affects the result of the second write, I edited the Client like this, but it still succeeds:
// Client
if conn, err := net.Dial("tcp", addr); err == nil {
_, err := conn.Write([]byte("hello"))
if err != nil {
fmt.Println(err)
conn.Close()
return
} else {
fmt.Println("ok")
}
// sleep 10 seconds and re-send
time.Sleep(10*time.Second)
b := make([]byte, 400000)
for i := range b {
b[i] = 'x'
}
n, err := conn.Write(b)
if err != nil {
fmt.Println(err)
conn.Close()
return
} else {
fmt.Println("ok", n)
}
// sleep 10 seconds and re-send
time.Sleep(10*time.Second)
} else {
panic(err)
}
And here is the screenshot:
attachment
There are several problems with your approach.
Sort-of a preface
The first one is that you do not wait for the server goroutine
to complete.
In Go, once main() exits for whatever reason,
all the other goroutines still running, if any, are simply
teared down forcibly.
You're trying to "synchronize" things using timers,
but this only works in toy situations, and even then it
does so only from time to time.
Hence let's fix your code first:
package main
import (
"fmt"
"log"
"net"
"time"
)
func main() {
addr := "127.0.0.1:8999"
tcpaddr, err := net.ResolveTCPAddr("tcp4", addr)
if err != nil {
log.Fatal(err)
}
listener, err := net.ListenTCP("tcp", tcpaddr)
if err != nil {
log.Fatal(err)
}
// Server
done := make(chan error)
go func(listener net.Listener, done chan<- error) {
for {
conn, err := listener.Accept()
if err != nil {
done <- err
return
}
go func(conn net.Conn) {
var buffer [1024]byte
n, err := conn.Read(buffer[:])
if err != nil {
log.Println(err)
} else {
log.Println(">", string(buffer[0:n]))
}
if err := conn.Close(); err != nil {
log.Println("error closing server conn:", err)
}
}(conn)
}
}(listener, done)
// Client
conn, err := net.Dial("tcp", addr)
if err != nil {
log.Fatal(err)
}
for i := 0; i < 2; i++ {
_, err := conn.Write([]byte("hello"))
if err != nil {
log.Println(err)
err = conn.Close()
if err != nil {
log.Println("error closing client conn:", err)
}
break
}
fmt.Println("ok")
time.Sleep(2 * time.Second)
}
// Shut the server down and wait for it to report back
err = listener.Close()
if err != nil {
log.Fatal("error closing listener:", err)
}
err = <-done
if err != nil {
log.Println("server returned:", err)
}
}
I've spilled a couple of minor fixes
like using log.Fatal (which is
log.Print + os.Exit(1)) instead of panicking,
removed useless else clauses to adhere to the coding standard of keeping the main
flow where it belongs, and lowered the client's timeout.
I have also added checking for possible errors Close on sockets may return.
The interesting part is that we now properly shut the server down by closing the listener and then waiting for the server goroutine to report back (unfortunately Go does not return an error of a custom type from net.Listener.Accept in this case so we can't really check that Accept exited because we've closed the listener).
Anyway, our goroutines are now properly synchronized, and there is
no undefined behaviour, so we can reason about how the code works.
Remaining problems
Some problems still remain.
The more glaring is you making wrong assumption that TCP preserves
message boundaries—that is, if you write "hello" to the client
end of the socket, the server reads back "hello".
This is not true: TCP considers both ends of the connection
as producing and consuming opaque streams of bytes.
This means, when the client writes "hello", the client's
TCP stack is free to deliver "he" and postpone sending "llo",
and the server's stack is free to yield "hell" to the read
call on the socket and only return "o" (and possibly some other
data) in a later read.
So, to make the code "real" you'd need to somehow introduce these
message boundaries into the protocol above TCP.
In this particular case the simplest approach would be either
using "messages" consisting of a fixed-length and agreed-upon
endianness prefix indicating the length of the following
data and then the string data itself.
The server would then use a sequence like
var msg [4100]byte
_, err := io.ReadFull(sock, msg[:4])
if err != nil { ... }
mlen := int(binary.BigEndian.Uint32(msg[:4]))
if mlen < 0 {
// handle error
}
if mlen == 0 {
// empty message; goto 1
}
_, err = io.ReadFull(sock, msg[5:5+mlen])
if err != nil { ... }
s := string(msg[5:5+mlen])
Another approach is to agree on that the messages do not contain
newlines and terminate each message with a newline
(ASCII LF, \n, 0x0a).
The server side would then use something like
a usual bufio.Scanner loop to get
full lines from the socket.
The remaining problem with your approach is to not dealing with
what Read on a socket returns: note that io.Reader.Read
(that's what sockets implement, among other things) is allowed
to return an error while having had read some data from the
underlying stream. In your toy example this might rightfully
be unimportant, but suppose that you're writing a wget-like
tool which is able to resume downloading of a file: even if
reading from the server returned some data and an error, you
have to deal with that returned chunk first and only then
handle the error.
Back to the problem at hand
The problem presented in the question, I beleive, happens simply because in your setup you hit some TCP buffering problem due to the tiny length of your messages.
On my box which runs Linux 4.9/amd64 two things reliably "fix"
the problem:
Sending messages of 4000 bytes in length: the second call
to Write "sees" the problem immediately.
Doing more Write calls.
For the former, try something like
msg := make([]byte, 4000)
for i := range msg {
msg[i] = 'x'
}
for {
_, err := conn.Write(msg)
...
and for the latter—something like
for {
_, err := conn.Write([]byte("hello"))
...
fmt.Println("ok")
time.Sleep(time.Second / 2)
}
(it's sensible to lower the pause between sending stuff in
both cases).
It's interesting to note that the former example hits the
write: connection reset by peer (ECONNRESET in POSIX)
error while the second one hits write: broken pipe
(EPIPE in POSIX).
This is because when we're sending in chunks worth 4k bytes,
some of the packets generated for the stream manage to become
"in flight" before the server's side of the connection manages
to propagate the information on its closure to the client,
and those packets hit an already closed socket and get rejected
with the RST TCP flag set.
In the second example an attempt to send another chunk of data
sees that the client side already knows that the connection
has been teared down and fails the sending without "touching
the wire".
TL;DR, the bottom line
Welcome to the wonderful world of networking. ;-)
I'd recommend buying a copy of "TCP/IP Illustrated",
read it and experiment.
TCP (and IP and other protocols above IP)
sometimes works not like people expect them to by applying
their "common sense".
I love the way Go handles I/O multiplexing internally which epoll and another mechanisms and schedules green threads (go-routine here) on its own giving the freedom to write synchronous code.
I know TCP sockets are non-blocking and read will give EAGAIN when no data is available. Given that, conn.Read(buffer) will detect this and blocks the go routine doing a connection read with no data available in the socket buffer. Is there a way to stop such go routine without closing the underlying connection. I am using a connection pool so closing the TCP connection won't make sense for me and want to return that connection back to the pool.
Here is the code to simulate such scenario:
func main() {
conn, _ := net.Dial("tcp", "127.0.0.1:9090")
// Spawning a go routine
go func(conn net.Conn) {
var message bytes.Buffer
for {
k := make([]byte, 255) // buffer
m, err := conn.Read(k) // blocks here
if err != nil {
if err != io.EOF {
fmt.Println("Read error : ", err)
} else {
fmt.Println("End of the file")
}
break // terminate loop if error
}
// converting bytes to string for printing
if m > 0 {
for _, b := range k {
message.WriteByte(b)
}
fmt.Println(message.String())
}
}
}(conn)
// prevent main from exiting
select {}
}
What are the other approaches can I take if it's not possible:
1) Call syscall.Read and handle this manually. In this case, I need a way to check if the socket is readable before calling syscall.Readotherwise I will end up wasting unnecessary CPU cycles. For my scenario, I think I can skip the event based polling thing and keep on calling syscall.Read as there always be data in my use case.
2) Any suggestions :)
func receive(conn net.TCPConn, kill <-chan struct{}) error {
// Spawn a goroutine to read from the connection.
data := make(chan []byte)
readErr := make(chan error)
go func() {
for {
b := make([]byte, 255)
_, err := conn.Read(b)
if err != nil {
readErr <- err
break
}
data <- b
}
}()
for {
select {
case b := <-data:
// Do something with `b`.
case err := <-readErr:
// Handle the error.
return err
case <-kill:
// Received kill signal, returning without closing the connection.
return nil
}
}
}
Send an empty struct to kill from another goroutine to stop receiving from the connection. Here's a program that stops receiving after a second:
kill := make(chan struct{})
go func() {
if err := receive(conn, kill); err != nil {
log.Fatal(err)
}
}()
time.Sleep(time.Second)
kill <- struct{}{}
This might not be exactly what you're looking for, because the reading goroutine would still be blocked on Read even after you send to kill. However, the goroutine that handles incoming reads would terminate.