Golang - net.Conn infinite loop on the same message - sockets

I'm pretty news in Golang and only use sockets in this langage for 2days. However, I'm not sure to understand something. I know in C, I used select() to know who wrote etc, but here, no one is writing until one send a message. After this message sent, my dialTCP uses it endlessly.
I think I missunderstood something about close() but I'm not sure it comes from here.. there is my code:
package dial
import (
"errors"
"encoding/json"
m "models"
"net"
"net/http"
"time"
"server"
)
type DialTCP struct {}
// tcpKeepAliveListener sets TCP keep-alive timeouts on accepted
// connections. It's used by ListenAndServe and ListenAndServeTLS so
// dead TCP connections (e.g. closing laptop mid-download) eventually
// go away.
type tcpKeepAliveListener struct {
*net.TCPListener
}
func (ln tcpKeepAliveListener) Accept() (c net.Conn, err error) {
tc, err := ln.AcceptTCP()
if err != nil {
return
}
tc.SetKeepAlive(true)
tc.SetKeepAlivePeriod(3 * time.Minute)
return tc, nil
}
func (dialTCP *DialTCP) ListenAndServe(addr string) error {
ln, err := net.Listen("tcp", addr)
if err != nil {
return err
}
return dialTCP.Serve(tcpKeepAliveListener{ln.(*net.TCPListener)})
}
func (dialTCP *DialTCP) Serve(l net.Listener) error {
defer l.Close()
for {
conn, e := l.Accept()
if e != nil {
return e
}
// you want to create server_conn here with buffers, channels and stuff
// to use async thread safe read/write from it
go dialTCP.serve_conn(conn)
}
}
func (dialTCP *DialTCP) serve_conn(conn net.Conn) error {
// var buf [512]byte
dec := json.NewDecoder(conn)
//read 1st message he sent, should be token to connect
var auth m.DialAuthentication
dec.Decode(&auth)
user := m.User{
UUID: auth.UUID,
}
ok, sb := server.IsConnected(user)
if ok == false {
json.NewEncoder(conn).Encode(sb)
return errors.New("User isn't connected.")
} else {
user.Conn = conn
}
//defer conn.Close()
var message m.DialMessageContainer
for {
dec.Decode(&message)
switch message.Type {
case ".....":
/* ....(message, user)
case "....":
....(message, user)
// case "...":*/
default:
json.NewEncoder(conn).Encode(m.StatusBack{Description: "Bad entry.", StatusId: http.StatusNotAcceptable})
}
//defer conn.Close()
}
}
I think everything is good before serv_conn(), but the error should comes from inside the for. I tried lot of things, but this for{} in Golang... Why does it have not any params/var such as C/C++/C#/Java?
for (int i = 0; i < 10; i++) {}
I'm lost about the closing of the Conn as well, so I continue to read tutorial, post and doc about how to use it.. Days pass whitout find anything

Related

Exec a function when (re)connected to MongoDB

I'm using https://github.com/mongodb/mongo-go-driver
Is there any chance to be notified when Go MongoDB client is connected or reconnected? I wish to log this event, but don't see a way, so currently reconnecting happens on the background and is invisible.
Trying according to the advice in a comment:
type wConn struct {
net.Conn
}
func (c *wConn) Close() error {
fmt.Println("closed")
return c.Conn.Close()
}
type wDialer struct {
net.Dialer
}
func NewDialer() *wDialer {
dialer := &wDialer{}
dialer.Timeout = 5 * time.Second
return dialer
}
func (d *wDialer) DialContext(ctx context.Context, network, address string) (net.Conn, error) {
c, err := ... // ??? How to get a connection?
// Like this?
// nd := &net.Dialer{}
// c, err := nd.DialContext(ctx, network, address)
if err != nil {
return nil, err
}
nc := &wConn{Conn: c}
return nc, nil
}
...
td := NewDialer()
client, err := mongo.NewClient(options.Client().ApplyURI("mongodb://127.0.0.1:27017").SetDialer(td))
Here the question near ??? is what is a way to get a Conn connection object?

Write on a closed net.Conn but returned nil error

Talk is cheap, so here we go the simple code:
package main
import (
"fmt"
"time"
"net"
)
func main() {
addr := "127.0.0.1:8999"
// Server
go func() {
tcpaddr, err := net.ResolveTCPAddr("tcp4", addr)
if err != nil {
panic(err)
}
listen, err := net.ListenTCP("tcp", tcpaddr)
if err != nil {
panic(err)
}
for {
if conn, err := listen.Accept(); err != nil {
panic(err)
} else if conn != nil {
go func(conn net.Conn) {
buffer := make([]byte, 1024)
n, err := conn.Read(buffer)
if err != nil {
fmt.Println(err)
} else {
fmt.Println(">", string(buffer[0 : n]))
}
conn.Close()
}(conn)
}
}
}()
time.Sleep(time.Second)
// Client
if conn, err := net.Dial("tcp", addr); err == nil {
for i := 0; i < 2; i++ {
_, err := conn.Write([]byte("hello"))
if err != nil {
fmt.Println(err)
conn.Close()
break
} else {
fmt.Println("ok")
}
// sleep 10 seconds and re-send
time.Sleep(10*time.Second)
}
} else {
panic(err)
}
}
Ouput:
> hello
ok
ok
The Client writes to the Server twice. After the first read, the Server closes the connection immediately, but the Client sleeps 10 seconds and then re-writes to the Server with the same already closed connection object(conn).
Why can the second write succeed (returned error is nil)?
Can anyone help?
PS:
In order to check if the buffering feature of the system affects the result of the second write, I edited the Client like this, but it still succeeds:
// Client
if conn, err := net.Dial("tcp", addr); err == nil {
_, err := conn.Write([]byte("hello"))
if err != nil {
fmt.Println(err)
conn.Close()
return
} else {
fmt.Println("ok")
}
// sleep 10 seconds and re-send
time.Sleep(10*time.Second)
b := make([]byte, 400000)
for i := range b {
b[i] = 'x'
}
n, err := conn.Write(b)
if err != nil {
fmt.Println(err)
conn.Close()
return
} else {
fmt.Println("ok", n)
}
// sleep 10 seconds and re-send
time.Sleep(10*time.Second)
} else {
panic(err)
}
And here is the screenshot:
attachment
There are several problems with your approach.
Sort-of a preface
The first one is that you do not wait for the server goroutine
to complete.
In Go, once main() exits for whatever reason,
all the other goroutines still running, if any, are simply
teared down forcibly.
You're trying to "synchronize" things using timers,
but this only works in toy situations, and even then it
does so only from time to time.
Hence let's fix your code first:
package main
import (
"fmt"
"log"
"net"
"time"
)
func main() {
addr := "127.0.0.1:8999"
tcpaddr, err := net.ResolveTCPAddr("tcp4", addr)
if err != nil {
log.Fatal(err)
}
listener, err := net.ListenTCP("tcp", tcpaddr)
if err != nil {
log.Fatal(err)
}
// Server
done := make(chan error)
go func(listener net.Listener, done chan<- error) {
for {
conn, err := listener.Accept()
if err != nil {
done <- err
return
}
go func(conn net.Conn) {
var buffer [1024]byte
n, err := conn.Read(buffer[:])
if err != nil {
log.Println(err)
} else {
log.Println(">", string(buffer[0:n]))
}
if err := conn.Close(); err != nil {
log.Println("error closing server conn:", err)
}
}(conn)
}
}(listener, done)
// Client
conn, err := net.Dial("tcp", addr)
if err != nil {
log.Fatal(err)
}
for i := 0; i < 2; i++ {
_, err := conn.Write([]byte("hello"))
if err != nil {
log.Println(err)
err = conn.Close()
if err != nil {
log.Println("error closing client conn:", err)
}
break
}
fmt.Println("ok")
time.Sleep(2 * time.Second)
}
// Shut the server down and wait for it to report back
err = listener.Close()
if err != nil {
log.Fatal("error closing listener:", err)
}
err = <-done
if err != nil {
log.Println("server returned:", err)
}
}
I've spilled a couple of minor fixes
like using log.Fatal (which is
log.Print + os.Exit(1)) instead of panicking,
removed useless else clauses to adhere to the coding standard of keeping the main
flow where it belongs, and lowered the client's timeout.
I have also added checking for possible errors Close on sockets may return.
The interesting part is that we now properly shut the server down by closing the listener and then waiting for the server goroutine to report back (unfortunately Go does not return an error of a custom type from net.Listener.Accept in this case so we can't really check that Accept exited because we've closed the listener).
Anyway, our goroutines are now properly synchronized, and there is
no undefined behaviour, so we can reason about how the code works.
Remaining problems
Some problems still remain.
The more glaring is you making wrong assumption that TCP preserves
message boundaries—that is, if you write "hello" to the client
end of the socket, the server reads back "hello".
This is not true: TCP considers both ends of the connection
as producing and consuming opaque streams of bytes.
This means, when the client writes "hello", the client's
TCP stack is free to deliver "he" and postpone sending "llo",
and the server's stack is free to yield "hell" to the read
call on the socket and only return "o" (and possibly some other
data) in a later read.
So, to make the code "real" you'd need to somehow introduce these
message boundaries into the protocol above TCP.
In this particular case the simplest approach would be either
using "messages" consisting of a fixed-length and agreed-upon
endianness prefix indicating the length of the following
data and then the string data itself.
The server would then use a sequence like
var msg [4100]byte
_, err := io.ReadFull(sock, msg[:4])
if err != nil { ... }
mlen := int(binary.BigEndian.Uint32(msg[:4]))
if mlen < 0 {
// handle error
}
if mlen == 0 {
// empty message; goto 1
}
_, err = io.ReadFull(sock, msg[5:5+mlen])
if err != nil { ... }
s := string(msg[5:5+mlen])
Another approach is to agree on that the messages do not contain
newlines and terminate each message with a newline
(ASCII LF, \n, 0x0a).
The server side would then use something like
a usual bufio.Scanner loop to get
full lines from the socket.
The remaining problem with your approach is to not dealing with
what Read on a socket returns: note that io.Reader.Read
(that's what sockets implement, among other things) is allowed
to return an error while having had read some data from the
underlying stream. In your toy example this might rightfully
be unimportant, but suppose that you're writing a wget-like
tool which is able to resume downloading of a file: even if
reading from the server returned some data and an error, you
have to deal with that returned chunk first and only then
handle the error.
Back to the problem at hand
The problem presented in the question, I beleive, happens simply because in your setup you hit some TCP buffering problem due to the tiny length of your messages.
On my box which runs Linux 4.9/amd64 two things reliably "fix"
the problem:
Sending messages of 4000 bytes in length: the second call
to Write "sees" the problem immediately.
Doing more Write calls.
For the former, try something like
msg := make([]byte, 4000)
for i := range msg {
msg[i] = 'x'
}
for {
_, err := conn.Write(msg)
...
and for the latter—something like
for {
_, err := conn.Write([]byte("hello"))
...
fmt.Println("ok")
time.Sleep(time.Second / 2)
}
(it's sensible to lower the pause between sending stuff in
both cases).
It's interesting to note that the former example hits the
write: connection reset by peer (ECONNRESET in POSIX)
error while the second one hits write: broken pipe
(EPIPE in POSIX).
This is because when we're sending in chunks worth 4k bytes,
some of the packets generated for the stream manage to become
"in flight" before the server's side of the connection manages
to propagate the information on its closure to the client,
and those packets hit an already closed socket and get rejected
with the RST TCP flag set.
In the second example an attempt to send another chunk of data
sees that the client side already knows that the connection
has been teared down and fails the sending without "touching
the wire".
TL;DR, the bottom line
Welcome to the wonderful world of networking. ;-)
I'd recommend buying a copy of "TCP/IP Illustrated",
read it and experiment.
TCP (and IP and other protocols above IP)
sometimes works not like people expect them to by applying
their "common sense".

Generic REST API Golang

Searching SO for Generic REST API Golang gives 0 results. Searching Google gives 2 results. So this question is maybe not correctly formulated or it is impossible to achieve in Golang.
My goal is to avoid repeating similar code over and over again. So I am trying to make the code in Golang as generic as possible. Write once, use many.
This is my first attempt to create a generic REST API for select in Golang. The code below gives almost what I want:
But the result is presented in the Terminal. I have no idea how to redirect the result to the browser.
package main
import (
"fmt"
"log"
"net/http"
"database/sql"
"time"
_ "github.com/lib/pq"
)
var db *sql.DB
func main() {
Connect()
http.HandleFunc("/", Query)
log.Fatal(http.ListenAndServe(":8080", nil))
}
func Connect() {
const (
host = "127.0.0.1"
port = 5432
user = "test"
password = "test"
dbname = "Test")
login := fmt.Sprintf("host=%s port=%d user=%s "+"password=%s dbname=%s sslmode=require", host, port, user, password, dbname)
var err error
db, err = sql.Open("postgres", login)
if err != nil {
log.Fatalln(err)
}
err = db.Ping()
if err != nil {
panic(err)
}
func Query(w http.ResponseWriter, r *http.Request) {
var query string
switch r.URL.String() {
case "/getuser":
query = "select * from getuser()"
case "/getco":
query = "select * from getco()"
case "/etc"
query = "select * from etc"
default:
query = ""
}
var err error
var rows *sql.Rows
rows, err = db.Query(query)
if err != nil {
http.Error(w, http.StatusText(500), 500)
return
}
defer rows.Close()
cols, err := rows.Columns()
vals := make([]interface{}, len(cols))
for i := 0; i < len(cols); i++ {
vals[i] = new(interface{})
if i != 0 {
fmt.Print("\t")
}
fmt.Print(cols[i])
}
fmt.Println()
for rows.Next() {
err = rows.Scan(vals...)
if err != nil {
fmt.Println(err)
continue
}
for i := 0; i < len(vals); i++ {
if i != 0 {
fmt.Print("\t")
}
printValue(vals[i].(*interface{}))
}
fmt.Println()
}
func printValue(pval *interface{}) {
switch v := (*pval).(type) {
case nil:
fmt.Print("NULL")
case bool:
if v {
fmt.Print("1")
} else {
fmt.Print("0")
}
case []byte:
fmt.Print(string(v))
case time.Time:
fmt.Print(v.Format("2006-01-02"))
default:
fmt.Print(v)
}
}
Every attempt to write to the browser gives various type of errors:
fmt.Printf("%s\n", vals...)
My questions are
How do I redirect the result to the browser?
Is there any better way to achieve this? (reuse generic code)
My recommendation would be to look at using existing packages like "mux" for calling REST APIs in browser. As a quick demo how you would do it as as follows:
your restapi.go cound have APIs as follows:
func SampleAPI(w http.ResponseWriter, r *http.Request) { //Assuming this is a POST request
var example SomeSruct
_ = json.NewDecoder(r.Body).Decode(&example) //Decode the POST body
result := someLogicFunction(example) //call your generic function
json.NewEncoder(w).Encode(result) //encode the result to pass it back to browser
}
Now say you write a main.go and you are using mux package here is an example of how you would call this
main.go
func main() {
router := mux.NewRouter()
router.HandleFunc("/testFunc",restapi.SampleAPI).Methods("POST") //This creates the route for your http request
handler := cros.Default().Handler(router) //You will need this if you plan to deploy it in a server and call it externally for testing locally you don't need this
log.Fatal(http.ListenAndServe(":8080", handler)) //Port that the router is listening to
}
Now note that you will have to import the "github.com/gorilla/mux" and the "github.com/rs/cors" packages to use these but this way you can create REST APIs whic can be accessed by te browser. Similarly you could create a GET method and use parameters which you can grab in your function and perform any logical step.
If you build and install the above code you can POST to localhost:8080/testFunc over http using any web app and get results i your browser. If you had a GET request you could directly type the Url in the browser and see the result.
write response with appropriahe HTTP hearers && status code
import "net/http"
func writeResponse(w http.ResponseWriter, contents []byte) {
w.Header().Set("Content-Type", "text/plain; charset=utf-8")
w.Header().Set("X-Content-Type-Options", "nosniff")
w.WriteHeader(http.StatusOK)
fmt.Fprintln(w, contents)
}
sounds a little unclear, sorry

too many open files in mgo go server

I'm getting these errors in the logs:
Accept error: accept tcp [::]:80: accept4: too many open files;
for a mongodb server on ubuntu, written in go using mgo. They start appearing after it's been running for about a day.
code:
package main
import (
"encoding/json"
"io"
"net/http"
"gopkg.in/mgo.v2/bson"
)
var (
Database *mgo.Database
)
func hello(w http.ResponseWriter, r *http.Request) {
io.WriteString(w, "hello")
}
func setTile(w http.ResponseWriter, r *http.Request) {
var requestJSON map[string]interface{}
err := json.NewDecoder(r.Body).Decode(&requestJSON)
if err != nil {
http.Error(w, err.Error(), 400)
return
}
collection := Database.C("tiles")
if requestJSON["tileId"] != nil {
query := bson.M{"tileId": requestJSON["tileId"]}
collection.RemoveAll(query)
collection.Insert(requestJSON)
w.WriteHeader(200)
w.Header().Set("Content-Type", "application/json")
js, _ := json.Marshal(map[string]string{"result": "ok"})
w.Write(js)
} else {
w.WriteHeader(200)
w.Header().Set("Content-Type", "application/json")
w.Write(js)
}
}
func getTile(w http.ResponseWriter, r *http.Request) {
var requestJSON map[string]interface{}
err := json.NewDecoder(r.Body).Decode(&requestJSON)
if err != nil {
http.Error(w, err.Error(), 400)
return
}
collection := Database.C("tiles")
var result []map[string]interface{}
if requestJSON["tileId"] != nil {
query := bson.M{"tileId": requestJSON["tileId"]}
collection.Find(query).All(&result)
}
if len(result) > 0 {
w.WriteHeader(200)
w.Header().Set("Content-Type", "application/json")
js, _ := json.Marshal(result[0])
w.Write(js)
} else {
w.WriteHeader(200)
w.Header().Set("Content-Type", "application/json")
js, _ := json.Marshal(map[string]string{"result": "tile id not found"})
w.Write(js)
}
}
func main() {
session, _ := mgo.Dial("localhost")
Database = session.DB("mapdb")
mux := http.NewServeMux()
mux.HandleFunc("/", hello)
mux.HandleFunc("/setTile", setTile)
mux.HandleFunc("/getTile", getTile)
http.ListenAndServe(":80", mux)
}
Is there something in there that needs closing? Or is it structured wrong in some way?
There seems to be lots of places to set the open file limits, so i'm not sure how to find out what the limits actually are. But it seems like increasing the limit isn't the problem anyway, surely something is being opened on every request and not closed.
This is not how you store and use a MongoDB connection in Go.
You have to store an mgo.Session, not an mgo.Database instance. And whenever you need to interact with the MongoDB, you acquire a copy or a clone of the session (e.g. with Session.Copy() or Session.Clone()), and you close it when you don't need it (preferable using a defer statement). This will ensure you don't leak connections.
You also religiously omit checking for errors, please don't do that. Whatever returns an error, do check it and act on it properly (the least you can do is print / log it).
So basically what you need to do is something like this:
var session *mgo.Session
func init() {
var err error
if session, err = mgo.Dial("localhost"); err != nil {
log.Fatal(err)
}
}
func someHandler(w http.ResponseWriter, r *http.Request) {
sess := session.Copy()
defer sess.Close() // Must close!
c := sess.DB("mapdb").C("tiles")
// Do something with the collection, e.g.
var tile bson.M
if err := c.FindId("someTileID").One(&result); err != nil {
// Tile does not exist, send back error, e.g.:
log.Printf("Tile with ID not found: %v, err: %v", "someTileID", err)
http.NotFound(w, r)
return
}
// Do something with tile
}
See related questions:
mgo - query performance seems consistently slow (500-650ms)
Concurrency in gopkg.in/mgo.v2 (Mongo, Go)
You are missing:
defer r.Body.Close()
Make sure it is used before return statement.

golang tcp socket does not send message after write immediately

My GO version is 1.1.1
the sever recieved messages after connection close, but NoDelay was setted.
Is there something wrong
addr, _ := net.ResolveTCPAddr("tcp", "localhost:5432")
conn, err := net.DialTCP("tcp", nil, addr)
defer conn.Close()
if err != nil {
fmt.Println("connect fail")
return
}
err = conn.SetNoDelay(true)
if err != nil {
fmt.Println(err.Error())
}
for {
var message string
_, err := fmt.Scanln(&message)
if err != nil && err.Error() != "unexpected newline" {
fmt.Println("input finished", err)
break
}
if message == "" {
fmt.Println("no input, end")
break
}
// message = fmt.Sprintf("%s\n",message)
//fmt.Fprintf(conn, message) // send immediately but following message won't send any more
conn.Write([]byte(message)) // won't send until connection close
}
There doesn't seem to be anything vitally wrong with your code so I'm guessing the error is on the server end.
If you create a local TCP server on port 5432 you can test this.
Try running the below server code and then test your client code against it. It just echos all received data to stdout.
package main
import (
"io"
"log"
"net"
"os"
)
func main() {
l, err := net.Listen("tcp", "localhost:5432")
if err != nil {
log.Fatal(err)
}
defer l.Close()
for {
conn, err := l.Accept()
if err != nil {
log.Fatal(err)
}
go func(c net.Conn) {
defer c.Close()
io.Copy(os.Stdout, c)
}(conn)
}
}
You should see each line sent to the client printed (without the newline) as soon as you hit enter.
the problem is on the server end.
func handleConnection(conn net.Conn) {
// I didn't put it in for loop
message, err := bufio.NewReader(conn).ReadString('\n')
}