I have zeromq: stable 4.1.4 installed using brew on MacOSX and have written a simple PUB/SUB program to test zeromq. But when I run the sample program using flags --bufsize > 5 (to use a buffer of size > 5MB) (go run go_zmq_pubsub.go --bufsize=6); it throws the following exception:
No buffer space available (tcp.cpp:69)
SIGABRT: abort
PC=0x7fff9911c286 m=0
signal arrived during cgo execution
Below is the program I used to test the zeromq4.x
package main
import (
"fmt"
"flag"
"strconv"
"sync"
log "github.com/Sirupsen/logrus"
zmq "github.com/pebbe/zmq4"
"time"
)
var _ = fmt.Println
func main(){
var port int
var bufsize int
flag.IntVar(&port, "port", 7676, "server's zmq tcp port")
flag.IntVar(&bufsize, "bufsize", 0, "socket kernel buffer size")
flag.Parse();
publisher, err := zmq.NewSocket(zmq.PUB)
if(err != nil) {
log.Fatal(err)
}
//set publisher kernel transmit buffer size
//convert into bytes
if err := publisher.SetSndbuf(bufsize * 1000000); err != nil {
log.Fatal(err)
}
defer publisher.Close()
publisher.Bind("tcp://*:" + strconv.Itoa(port))
//SETUP subscriber
subscriber, err := zmq.NewSocket(zmq.SUB)
if(err != nil) {
log.Fatal(err)
}
//set subscriber kernel receive buffer size
if err := subscriber.SetRcvbuf(bufsize * 1000000); err != nil {
log.Fatal(err)
}
defer subscriber.Close()
subscriber.Connect("tcp://127.0.0.1:" + strconv.Itoa(port))
subscriber.SetSubscribe("")
var wg sync.WaitGroup
wg.Add(2)
idx := 0
go func(wg *sync.WaitGroup) {
//start streaming messages
ticker := time.NewTicker(1 * time.Second)
go func() {
for {
select {
case <-ticker.C:
_, err = publisher.Send("PKG:"+strconv.Itoa(idx), 0)
idx++;
if(err != nil) {
log.Error(err)
}
}
}
}()
}(&wg)
//receiver
go func(wg *sync.WaitGroup) {
go func(){
for {
payload, err := subscriber.Recv(0)
_ = payload
if err != nil {
log.Error(err)
break
}
//now sending into worker pool
log.Info("RECEIVE:" + payload)
}
}()
}(&wg)
wg.Wait()
}
On Centos7 with lib-zeromq built from source, the above code works without problem.
Not sure if it's due to libzeromq or the OS itself.
Thanks.
A buffer size of > 5MB is pointless. Anything beyond the bandwidth-delay product of the link concerned is wasted space.
Moderate your requirements.
Related
I am new to programming Golang Sockets. When I try to send one message from client to server, it is working perfectly. However, when I try to send 10 consecutive messages, I get an error. Any clues/keywords to search for. Please find enclosed a sample code.
Server.go
package main
import (
"encoding/gob"
"fmt"
"net"
"os"
)
func main() {
tcpAddr, err := net.ResolveTCPAddr("tcp4", ":5555")
checkError("ResolveTCPAddr", err)
listener, err := net.ListenTCP("tcp", tcpAddr)
checkError("ListenTCP", err)
conn, err := listener.Accept()
checkError("Accept", err)
for i := 0; i < 10; i++ {
var s string
dec := gob.NewDecoder(conn)
err = dec.Decode(&s)
checkError("Decode", err)
fmt.Println(s)
}
}
func checkError(info string, err error) {
if err != nil {
fmt.Fprintf(os.Stderr, info+": Run - Fatal error: %s\n", err.Error())
os.Exit(1)
}
}
Client.go
package main
import (
"encoding/gob"
"fmt"
"net"
"os"
)
func main() {
tcpAddr, err := net.ResolveTCPAddr("tcp4", ":5555")
checkError("ResolveTCPAddr", err)
conn, err := net.DialTCP("tcp", nil, tcpAddr)
checkError("DialTCP", err)
for i := 0; i < 10; i++ {
enc := gob.NewEncoder(conn)
err = enc.Encode("test")
checkError("Encode", err)
}
}
func checkError(info string, err error) {
if err != nil {
fmt.Fprintf(os.Stderr, info+": Run - Fatal error: %s\n", err.Error())
os.Exit(1)
}
}
SCREEN:
test
test
test
test
test
Decode: Run - Fatal error: EOF
exit status 1
The problem is that a decoder buffers data from the underlying reader and that buffered data can include data from a later message in the stream. The buffered data is discarded when the application discards the decoder. A later decoder returns an error because it is reading an incomplete message.
There's an easy fix to this problem. The gob package is designed to read and write streams of values. Create the encoder and decoder outside of the loop and let the package handle the message framing.
enc := gob.NewEncoder(conn)
for i := 0; i < 10; i++ {
err = enc.Encode("test")
checkError("Encode", err)
}
dec := gob.NewDecoder(conn)
for i := 0; i < 10; i++ {
var s string
err = dec.Decode(&s)
checkError("Decode", err)
fmt.Println(s)
}
If for some reason you must create the encoder and decoder inside the loop, then the application must implement message framing to prevent the decoder from reading more than a single value. One way to frame the messages is to have the client write a length prefix before the gob encoded value. The server reads the length and then limits the decoder to reading that number of bytes.
I am trying to create a TCP server and client using Golang where I am able to set the Type of Service field in the IP header in order to prioritise different traffic flows.
The client and servers are able to communicate but I can not figure out how to set the ToS field.
I have tried using the ipv4 Golang package with the method described here: https://godoc.org/golang.org/x/net/ipv4#NewConn
A simplified server example:
func main () {
ln, err := net.Listen("tcp4", "192.168.0.20:1024")
if err != nil {
// error handling
}
defer ln.Close()
for {
c, err := ln.Accept()
if err != nil {
// error handling
}
go func(c net.Conn) {
defer c.Close()
if err := ipv4.NewConn(c).SetTOS(0x28); err != nil {
fmt.Println("Error: ", err.Error())
}
}(c)
}
And the corresponding client (also simplified)
func main () {
conn, err := net.Dial("tcp4", "192.168.0.20:1024")
if err != nil {
fmt.Println(err)
}
for {
writer := bufio.NewWriter(conn)
// Create "packet"
Data := make([]byte, 1200)
endLine := "\r\n"
//Set packetLength
length := strconv.FormatInt(int64(1200), 10)
copy(Data[0:], length)
//Set ID
idString := strconv.FormatInt(int64(1), 10)
if strings.Contains(idString, "\r") || strings.Contains(idString, "\n") || strings.Contains(idString, "\r\n") {
fmt.Println("This is gonna result in an error in the id string.")
}
idbuf := []byte(idString)
copy(Data[15:], idbuf)
//Set timestamp
timestamp0 := time.Now().UnixNano()
timestampString := strconv.FormatInt(timestamp0, 10)
if strings.Contains(timestampString, "\r") || strings.Contains(timestampString, "\n") || strings.Contains(timestampString, "\r\n") {
fmt.Println("This is gonna result in an error in the timestamp string.")
}
buf := []byte(timestampString)
copy(Data[50:], buf)
copy(Data[int(1200)-2:], endLine)
if len(Data) != int(1200) {
fmt.Println("This is also gonna be an error. Length is: ", len(Data))
}
//Send the data and flush the writer
writer.Write(Data)
writer.Flush()
}
//time.Sleep(1*time.Nanosecond)
}
I have also tried creating my own dialer with a control function that passes a syscall in order to set the socket like this:
dialer := &net.Dialer{
Timeout: 5 * time.Second,
Deadline: time.Time{},
LocalAddr: tcpAddr,
DualStack: false,
FallbackDelay: 0,
KeepAlive: 0,
Resolver: nil,
Control: highPrio,
}
func highPrio(network, address string, c syscall.RawConn) error {
return c.Control(func(fd uintptr) {
// set the socket options
err := syscall.SetsockoptInt(syscall.Handle(fd), syscall.IPPROTO_IP, syscall.IP_TOS, 128)
if err != nil {
log.Println("setsocketopt: ", err)
}
})
I am verifying that it does not work by inspecting the traffic with Wireshark and am using Windows 10 Pro as my OS.
I am try you ToS set method at Dial() with golang 1.15.5 and its worked:
dialer := net.Dialer{
Timeout: this.TcpWaitConnectTimeout,
}
dialer.Control = func(network, address string, c syscall.RawConn) error {
var err error
c.Control(func(fd uintptr) {
err = syscall.SetsockoptInt(int(fd), syscall.IPPROTO_IP, syscall.IP_TOS, 0x80)
})
return err
}
c, err := dialer.Dial("tcp", this.serverAddr)
tcpdump show me right ToS
Talk is cheap, so here we go the simple code:
package main
import (
"fmt"
"time"
"net"
)
func main() {
addr := "127.0.0.1:8999"
// Server
go func() {
tcpaddr, err := net.ResolveTCPAddr("tcp4", addr)
if err != nil {
panic(err)
}
listen, err := net.ListenTCP("tcp", tcpaddr)
if err != nil {
panic(err)
}
for {
if conn, err := listen.Accept(); err != nil {
panic(err)
} else if conn != nil {
go func(conn net.Conn) {
buffer := make([]byte, 1024)
n, err := conn.Read(buffer)
if err != nil {
fmt.Println(err)
} else {
fmt.Println(">", string(buffer[0 : n]))
}
conn.Close()
}(conn)
}
}
}()
time.Sleep(time.Second)
// Client
if conn, err := net.Dial("tcp", addr); err == nil {
for i := 0; i < 2; i++ {
_, err := conn.Write([]byte("hello"))
if err != nil {
fmt.Println(err)
conn.Close()
break
} else {
fmt.Println("ok")
}
// sleep 10 seconds and re-send
time.Sleep(10*time.Second)
}
} else {
panic(err)
}
}
Ouput:
> hello
ok
ok
The Client writes to the Server twice. After the first read, the Server closes the connection immediately, but the Client sleeps 10 seconds and then re-writes to the Server with the same already closed connection object(conn).
Why can the second write succeed (returned error is nil)?
Can anyone help?
PS:
In order to check if the buffering feature of the system affects the result of the second write, I edited the Client like this, but it still succeeds:
// Client
if conn, err := net.Dial("tcp", addr); err == nil {
_, err := conn.Write([]byte("hello"))
if err != nil {
fmt.Println(err)
conn.Close()
return
} else {
fmt.Println("ok")
}
// sleep 10 seconds and re-send
time.Sleep(10*time.Second)
b := make([]byte, 400000)
for i := range b {
b[i] = 'x'
}
n, err := conn.Write(b)
if err != nil {
fmt.Println(err)
conn.Close()
return
} else {
fmt.Println("ok", n)
}
// sleep 10 seconds and re-send
time.Sleep(10*time.Second)
} else {
panic(err)
}
And here is the screenshot:
attachment
There are several problems with your approach.
Sort-of a preface
The first one is that you do not wait for the server goroutine
to complete.
In Go, once main() exits for whatever reason,
all the other goroutines still running, if any, are simply
teared down forcibly.
You're trying to "synchronize" things using timers,
but this only works in toy situations, and even then it
does so only from time to time.
Hence let's fix your code first:
package main
import (
"fmt"
"log"
"net"
"time"
)
func main() {
addr := "127.0.0.1:8999"
tcpaddr, err := net.ResolveTCPAddr("tcp4", addr)
if err != nil {
log.Fatal(err)
}
listener, err := net.ListenTCP("tcp", tcpaddr)
if err != nil {
log.Fatal(err)
}
// Server
done := make(chan error)
go func(listener net.Listener, done chan<- error) {
for {
conn, err := listener.Accept()
if err != nil {
done <- err
return
}
go func(conn net.Conn) {
var buffer [1024]byte
n, err := conn.Read(buffer[:])
if err != nil {
log.Println(err)
} else {
log.Println(">", string(buffer[0:n]))
}
if err := conn.Close(); err != nil {
log.Println("error closing server conn:", err)
}
}(conn)
}
}(listener, done)
// Client
conn, err := net.Dial("tcp", addr)
if err != nil {
log.Fatal(err)
}
for i := 0; i < 2; i++ {
_, err := conn.Write([]byte("hello"))
if err != nil {
log.Println(err)
err = conn.Close()
if err != nil {
log.Println("error closing client conn:", err)
}
break
}
fmt.Println("ok")
time.Sleep(2 * time.Second)
}
// Shut the server down and wait for it to report back
err = listener.Close()
if err != nil {
log.Fatal("error closing listener:", err)
}
err = <-done
if err != nil {
log.Println("server returned:", err)
}
}
I've spilled a couple of minor fixes
like using log.Fatal (which is
log.Print + os.Exit(1)) instead of panicking,
removed useless else clauses to adhere to the coding standard of keeping the main
flow where it belongs, and lowered the client's timeout.
I have also added checking for possible errors Close on sockets may return.
The interesting part is that we now properly shut the server down by closing the listener and then waiting for the server goroutine to report back (unfortunately Go does not return an error of a custom type from net.Listener.Accept in this case so we can't really check that Accept exited because we've closed the listener).
Anyway, our goroutines are now properly synchronized, and there is
no undefined behaviour, so we can reason about how the code works.
Remaining problems
Some problems still remain.
The more glaring is you making wrong assumption that TCP preserves
message boundaries—that is, if you write "hello" to the client
end of the socket, the server reads back "hello".
This is not true: TCP considers both ends of the connection
as producing and consuming opaque streams of bytes.
This means, when the client writes "hello", the client's
TCP stack is free to deliver "he" and postpone sending "llo",
and the server's stack is free to yield "hell" to the read
call on the socket and only return "o" (and possibly some other
data) in a later read.
So, to make the code "real" you'd need to somehow introduce these
message boundaries into the protocol above TCP.
In this particular case the simplest approach would be either
using "messages" consisting of a fixed-length and agreed-upon
endianness prefix indicating the length of the following
data and then the string data itself.
The server would then use a sequence like
var msg [4100]byte
_, err := io.ReadFull(sock, msg[:4])
if err != nil { ... }
mlen := int(binary.BigEndian.Uint32(msg[:4]))
if mlen < 0 {
// handle error
}
if mlen == 0 {
// empty message; goto 1
}
_, err = io.ReadFull(sock, msg[5:5+mlen])
if err != nil { ... }
s := string(msg[5:5+mlen])
Another approach is to agree on that the messages do not contain
newlines and terminate each message with a newline
(ASCII LF, \n, 0x0a).
The server side would then use something like
a usual bufio.Scanner loop to get
full lines from the socket.
The remaining problem with your approach is to not dealing with
what Read on a socket returns: note that io.Reader.Read
(that's what sockets implement, among other things) is allowed
to return an error while having had read some data from the
underlying stream. In your toy example this might rightfully
be unimportant, but suppose that you're writing a wget-like
tool which is able to resume downloading of a file: even if
reading from the server returned some data and an error, you
have to deal with that returned chunk first and only then
handle the error.
Back to the problem at hand
The problem presented in the question, I beleive, happens simply because in your setup you hit some TCP buffering problem due to the tiny length of your messages.
On my box which runs Linux 4.9/amd64 two things reliably "fix"
the problem:
Sending messages of 4000 bytes in length: the second call
to Write "sees" the problem immediately.
Doing more Write calls.
For the former, try something like
msg := make([]byte, 4000)
for i := range msg {
msg[i] = 'x'
}
for {
_, err := conn.Write(msg)
...
and for the latter—something like
for {
_, err := conn.Write([]byte("hello"))
...
fmt.Println("ok")
time.Sleep(time.Second / 2)
}
(it's sensible to lower the pause between sending stuff in
both cases).
It's interesting to note that the former example hits the
write: connection reset by peer (ECONNRESET in POSIX)
error while the second one hits write: broken pipe
(EPIPE in POSIX).
This is because when we're sending in chunks worth 4k bytes,
some of the packets generated for the stream manage to become
"in flight" before the server's side of the connection manages
to propagate the information on its closure to the client,
and those packets hit an already closed socket and get rejected
with the RST TCP flag set.
In the second example an attempt to send another chunk of data
sees that the client side already knows that the connection
has been teared down and fails the sending without "touching
the wire".
TL;DR, the bottom line
Welcome to the wonderful world of networking. ;-)
I'd recommend buying a copy of "TCP/IP Illustrated",
read it and experiment.
TCP (and IP and other protocols above IP)
sometimes works not like people expect them to by applying
their "common sense".
I have a tcp server and a client, the server does the following
func providerCallback(conn net.Conn) {
reader := bufio.NewReader(conn)
var err error
for {
lenbyte, _ := reader.Peek(4)
reader.Discard(4)
slen := int(binary.BigEndian.Uint32(lenbyte))
data, err = reader.Peek(slen)
process(data)
reader.Discard(slen)
}
}
The client seems to send packet faster than process can deal with, therefore I'd like to buffer the requests in bufio and process later.
However, as the size of bufio is fixed(4096, even though I can increase it, it is still fixed), which means I can't manually Reset it because there might be a packet cutting of in the end of bufio, as follows
|normal data... [First 20 bytes of packet P] | [the rest of packet P]
|------------------- size of bufio ------------------|
How can I splice packet that is cut off, and reuse the bufio for later packets?
For example,
import (
"bufio"
"encoding/binary"
"io"
"net"
)
func providerCallback(conn net.Conn) error {
rdr := bufio.NewReader(conn)
data := make([]byte, 0, 4*1024)
for {
n, err := io.ReadFull(rdr, data[:4])
data = data[:n]
if err != nil {
if err == io.EOF {
break
}
return err
}
dataLen := binary.BigEndian.Uint32(data)
if uint64(dataLen) > uint64(cap(data)) {
data = make([]byte, 0, dataLen)
}
n, err = io.ReadFull(rdr, data[:dataLen])
data = data[:n]
if err != nil {
return err
}
process(data)
}
return nil
}
func process([]byte) {}
I'm developing a fast dns client in go just to mess around with But I'm facing troubles at the time of reading from server responses cause it never arrives and I know it actually did because I have WireShark open and it read the packet.
Here is the code sample(8.8.8.8 is Google DNS and the hex msg is a valid DNS query):
package main
import (
"fmt"
"net"
"encoding/hex"
"bufio"
)
func CheckError(err error) {
if err != nil {
fmt.Println("Error: " , err)
}
}
func main() {
Conn, err := net.Dial("udp", "8.8.8.8:53")
CheckError(err)
defer Conn.Close()
msg, _ := hex.DecodeString("5ab9010000010000000000001072312d2d2d736e2d68357137646e65650b676f6f676c65766964656f03636f6d0000010001")
scanner := bufio.NewScanner(Conn)
buf := []byte(msg)
_, err1 := Conn.Write(buf)
if err1 != nil {
fmt.Println(msg, err1)
}
for scanner.Scan() {
fmt.Println(scanner.Bytes())
}
}
Here you have the proof that it actually arrives:
WireShark Screen Capture
I've testes reading directly from conn with:
func main() {
Conn, err := net.Dial("udp", "8.8.8.8:53")
CheckError(err)
defer Conn.Close()
msg, _ := hex.DecodeString("5ab9010000010000000000001072312d2d2d736e2d68357137646e65650b676f6f676c65766964656f03636f6d0000010001")
buf := []byte(msg)
_, err1 := Conn.Write(buf)
if err1 != nil {
fmt.Println(msg, err1)
}
Reader(Conn)
}
func Reader(conn net.Conn) {
var buf []byte
for {
conn.Read(buf)
fmt.Println(buf)
}
}
You can't use bufio around a UDP connection. UDP is not a stream oriented protocol, so you need to differentiate the individual datagrams yourself, and avoid partial reads to prevent data loss.
In order to read from an io.Reader, you must have space allocated to read into, and you need to use the bytes read value returned from the Read operation. Your example could be reduced to:
conn, err := net.Dial("udp", "8.8.8.8:53")
if err != nil {
log.Fatal(err)
}
defer conn.Close()
msg, _ := base64.RawStdEncoding.DecodeString("WrkBAAABAAAAAAAAEHIxLS0tc24taDVxN2RuZWULZ29vZ2xldmlkZW8DY29tAAABAAE")
resp := make([]byte, 512)
conn.Write(msg)
n, err := conn.Read(resp)
if err != nil {
log.Fatal(err)
}
fmt.Printf("%q\n", resp[:n])