I'm using https://github.com/mongodb/mongo-go-driver
Is there any chance to be notified when Go MongoDB client is connected or reconnected? I wish to log this event, but don't see a way, so currently reconnecting happens on the background and is invisible.
Trying according to the advice in a comment:
type wConn struct {
net.Conn
}
func (c *wConn) Close() error {
fmt.Println("closed")
return c.Conn.Close()
}
type wDialer struct {
net.Dialer
}
func NewDialer() *wDialer {
dialer := &wDialer{}
dialer.Timeout = 5 * time.Second
return dialer
}
func (d *wDialer) DialContext(ctx context.Context, network, address string) (net.Conn, error) {
c, err := ... // ??? How to get a connection?
// Like this?
// nd := &net.Dialer{}
// c, err := nd.DialContext(ctx, network, address)
if err != nil {
return nil, err
}
nc := &wConn{Conn: c}
return nc, nil
}
...
td := NewDialer()
client, err := mongo.NewClient(options.Client().ApplyURI("mongodb://127.0.0.1:27017").SetDialer(td))
Here the question near ??? is what is a way to get a Conn connection object?
Related
I am trying to create a TCP server and client using Golang where I am able to set the Type of Service field in the IP header in order to prioritise different traffic flows.
The client and servers are able to communicate but I can not figure out how to set the ToS field.
I have tried using the ipv4 Golang package with the method described here: https://godoc.org/golang.org/x/net/ipv4#NewConn
A simplified server example:
func main () {
ln, err := net.Listen("tcp4", "192.168.0.20:1024")
if err != nil {
// error handling
}
defer ln.Close()
for {
c, err := ln.Accept()
if err != nil {
// error handling
}
go func(c net.Conn) {
defer c.Close()
if err := ipv4.NewConn(c).SetTOS(0x28); err != nil {
fmt.Println("Error: ", err.Error())
}
}(c)
}
And the corresponding client (also simplified)
func main () {
conn, err := net.Dial("tcp4", "192.168.0.20:1024")
if err != nil {
fmt.Println(err)
}
for {
writer := bufio.NewWriter(conn)
// Create "packet"
Data := make([]byte, 1200)
endLine := "\r\n"
//Set packetLength
length := strconv.FormatInt(int64(1200), 10)
copy(Data[0:], length)
//Set ID
idString := strconv.FormatInt(int64(1), 10)
if strings.Contains(idString, "\r") || strings.Contains(idString, "\n") || strings.Contains(idString, "\r\n") {
fmt.Println("This is gonna result in an error in the id string.")
}
idbuf := []byte(idString)
copy(Data[15:], idbuf)
//Set timestamp
timestamp0 := time.Now().UnixNano()
timestampString := strconv.FormatInt(timestamp0, 10)
if strings.Contains(timestampString, "\r") || strings.Contains(timestampString, "\n") || strings.Contains(timestampString, "\r\n") {
fmt.Println("This is gonna result in an error in the timestamp string.")
}
buf := []byte(timestampString)
copy(Data[50:], buf)
copy(Data[int(1200)-2:], endLine)
if len(Data) != int(1200) {
fmt.Println("This is also gonna be an error. Length is: ", len(Data))
}
//Send the data and flush the writer
writer.Write(Data)
writer.Flush()
}
//time.Sleep(1*time.Nanosecond)
}
I have also tried creating my own dialer with a control function that passes a syscall in order to set the socket like this:
dialer := &net.Dialer{
Timeout: 5 * time.Second,
Deadline: time.Time{},
LocalAddr: tcpAddr,
DualStack: false,
FallbackDelay: 0,
KeepAlive: 0,
Resolver: nil,
Control: highPrio,
}
func highPrio(network, address string, c syscall.RawConn) error {
return c.Control(func(fd uintptr) {
// set the socket options
err := syscall.SetsockoptInt(syscall.Handle(fd), syscall.IPPROTO_IP, syscall.IP_TOS, 128)
if err != nil {
log.Println("setsocketopt: ", err)
}
})
I am verifying that it does not work by inspecting the traffic with Wireshark and am using Windows 10 Pro as my OS.
I am try you ToS set method at Dial() with golang 1.15.5 and its worked:
dialer := net.Dialer{
Timeout: this.TcpWaitConnectTimeout,
}
dialer.Control = func(network, address string, c syscall.RawConn) error {
var err error
c.Control(func(fd uintptr) {
err = syscall.SetsockoptInt(int(fd), syscall.IPPROTO_IP, syscall.IP_TOS, 0x80)
})
return err
}
c, err := dialer.Dial("tcp", this.serverAddr)
tcpdump show me right ToS
I'm having an issue with Gorm / Psql where my database connection get automatically closed.
I never call defer dbInstance.Close() in main.go (not anymore for now, I've removed it, since that's the only place in my code where I felt the connection could be wrongfully closed) nor was it ever anywhere else.
The way I'm initializing my db is with a "db" package that looks like this:
package db
import (
"fmt"
"github.com/jinzhu/gorm"
_ "github.com/jinzhu/gorm/dialects/postgres"
)
var DbInstance *gorm.DB
func Init() *gorm.DB {
if DbInstance != nil {
return DbInstance
}
fmt.Println("Opening Db")
db, err := gorm.Open("postgres", "host=localhost port=5432 user=admin dbname=dbname sslmode=disable password=")
if err != nil {
fmt.Println(err)
panic("failed to connect database")
}
return db
}
then I call db.Init() in my main.go and then only call the db from "db.dbInstance" from the rest of my program.
As I've previously mentioned I used to call defer db.DbInstance.Close() from main.go but have tried removing it to see if it fixed the issue but it didn't.
What's strange is that the db connection will work for hours and hours in many different calls/function but always end up closing at some point.
From what i understand it should work :
gorm.Open() uses https://golang.org/pkg/database/sql/, which is
threadsafe and handles connection pooling for you. Don't call
gorm.Open() every request. Call it once when setting up your
application, and make sure you use defer db.Close() so connections are
cleanly ended when your program exits.
Lastly I need to add that it seems (i'm not 100% sure) that it's closing after I do batch inserts but again the .Close() function is never called, anywhere in my program.
I'm a bit lost as to what could be happening? Garbage collector (doesn't make sense the var is global)? psql driver closing in the background? Configuration issue?
I'm adding the batch function for reference just in case:
func InsertWithPostGresLimitSizeV2(DB *gorm.DB, array []interface{}) {
if len(array) == 0 {
return
}
numberOfParams := len(DB.NewScope(array[0]).Fields())
// postgres is limited to 65535 params.
maxStructPerBulk := int(65535 / numberOfParams)
currentIndex := 0
if len(array) > maxStructPerBulk {
for len(array) > currentIndex {
if (maxStructPerBulk + currentIndex) < len(array) {
slice := array[currentIndex:(currentIndex + maxStructPerBulk)]
currentIndex += maxStructPerBulk
_, err := DB.BatchInsert(slice)
log.Println(err)
} else {
slice := array[currentIndex:len(array)]
currentIndex = len(array)
_, err := DB.BatchInsert(slice)
log.Println(err)
}
}
} else {
_, err := DB.BatchInsert(array)
log.Println(err)
}
}
func BatchInsert(db *gorm.DB,objArr []interface{}) (int64, error) {
if len(objArr) == 0 {
return 0, errors.New("insert a slice length of 0")
}
mainObj := objArr[0]
mainScope := db.NewScope(mainObj)
mainFields := mainScope.Fields()
quoted := make([]string, 0, len(mainFields))
for i := range mainFields {
if (mainFields[i].IsPrimaryKey && mainFields[i].IsBlank) || (mainFields[i].IsIgnored) {
continue
}
quoted = append(quoted, mainScope.Quote(mainFields[i].DBName))
}
placeholdersArr := make([]string, 0, len(objArr))
for _, obj := range objArr {
scope := db.NewScope(obj)
fields := scope.Fields()
placeholders := make([]string, 0, len(fields))
for i := range fields {
if (fields[i].IsPrimaryKey && fields[i].IsBlank) || (fields[i].IsIgnored) {
continue
}
var vars interface{}
if (fields[i].Name == "CreatedAt" || fields[i].Name == "UpdatedAt") && fields[i].IsBlank {
vars = gorm.NowFunc()
} else {
vars = fields[i].Field.Interface()
}
placeholders = append(placeholders, mainScope.AddToVars(vars))
}
placeholdersStr := "(" + strings.Join(placeholders, ", ") + ")"
placeholdersArr = append(placeholdersArr, placeholdersStr)
mainScope.SQLVars = append(mainScope.SQLVars, scope.SQLVars...)
}
mainScope.Raw(fmt.Sprintf("INSERT INTO %s (%s) VALUES %s",
mainScope.QuotedTableName(),
strings.Join(quoted, ", "),
strings.Join(placeholdersArr, ", "),
))
if err := mainScope.Exec().DB().Error; err != nil {
return 0, err
}
return mainScope.DB().RowsAffected, nil
}
On last thing is that I was thinking of " fixing " the issue by calling my db through but the ping would slow each of my calls:
func getDb() *gorm.DB {
err := DbInstance.DB().Ping()
if err != nil {
fmt.Println("Connection to db closed opening a new one")
return Init()
}
return DbInstance
}
You can global-search DbInstance.Close() to ensure not ever call it to close it yourself.
If not, you kan set db timeout for longer and raise amount of idle db connections.
At last, It's most important to support auto-reconnecting db dataSource.
Here is part of my auto-reconnecting part you might refer to:
var DB *gorm.DB
func init() {
dbConfig = fmt.Sprintf("host=%s user=%s dbname=%s sslmode=%s password=%s",
"localhost",
"postgres",
"dbname",
"disable",
"password",
)
db, err := gorm.Open("postgres",
dbConfig,
)
db.SingularTable(true)
db.LogMode(true)
db.DB().SetConnMaxLifetime(10 * time.Second)
db.DB().SetMaxIdleConns(30)
DB = db
// auto-connect，ping per 60s, re-connect on fail or error with intervels 3s, 3s, 15s, 30s, 60s, 60s ...
go func(dbConfig string) {
var intervals = []time.Duration{3 * time.Second, 3 * time.Second, 15 * time.Second, 30 * time.Second, 60 * time.Second,
}
for {
time.Sleep(60 * time.Second)
if e := DB.DB().Ping(); e != nil {
L:
for i := 0; i < len(intervals); i++ {
e2 := RetryHandler(3, func() (bool, error) {
var e error
DB, e = gorm.Open("postgres", dbConfig)
if e != nil {
return false, errorx.Wrap(e)
}
return true, nil
})
if e2 != nil {
fmt.Println(e.Error())
time.Sleep(intervals[i])
if i == len(intervals)-1 {
i--
}
continue
}
break L
}
}
}
}(dbConfig)
}
By the way:
// Try f() n times on fail and one time on success
func RetryHandler(n int, f func() (bool, error)) error {
ok, er := f()
if ok && er == nil {
return nil
}
if n-1 > 0 {
return RetryHandler(n-1, f)
}
return er
}
I'm getting these errors in the logs:
Accept error: accept tcp [::]:80: accept4: too many open files;
for a mongodb server on ubuntu, written in go using mgo. They start appearing after it's been running for about a day.
code:
package main
import (
"encoding/json"
"io"
"net/http"
"gopkg.in/mgo.v2/bson"
)
var (
Database *mgo.Database
)
func hello(w http.ResponseWriter, r *http.Request) {
io.WriteString(w, "hello")
}
func setTile(w http.ResponseWriter, r *http.Request) {
var requestJSON map[string]interface{}
err := json.NewDecoder(r.Body).Decode(&requestJSON)
if err != nil {
http.Error(w, err.Error(), 400)
return
}
collection := Database.C("tiles")
if requestJSON["tileId"] != nil {
query := bson.M{"tileId": requestJSON["tileId"]}
collection.RemoveAll(query)
collection.Insert(requestJSON)
w.WriteHeader(200)
w.Header().Set("Content-Type", "application/json")
js, _ := json.Marshal(map[string]string{"result": "ok"})
w.Write(js)
} else {
w.WriteHeader(200)
w.Header().Set("Content-Type", "application/json")
w.Write(js)
}
}
func getTile(w http.ResponseWriter, r *http.Request) {
var requestJSON map[string]interface{}
err := json.NewDecoder(r.Body).Decode(&requestJSON)
if err != nil {
http.Error(w, err.Error(), 400)
return
}
collection := Database.C("tiles")
var result []map[string]interface{}
if requestJSON["tileId"] != nil {
query := bson.M{"tileId": requestJSON["tileId"]}
collection.Find(query).All(&result)
}
if len(result) > 0 {
w.WriteHeader(200)
w.Header().Set("Content-Type", "application/json")
js, _ := json.Marshal(result[0])
w.Write(js)
} else {
w.WriteHeader(200)
w.Header().Set("Content-Type", "application/json")
js, _ := json.Marshal(map[string]string{"result": "tile id not found"})
w.Write(js)
}
}
func main() {
session, _ := mgo.Dial("localhost")
Database = session.DB("mapdb")
mux := http.NewServeMux()
mux.HandleFunc("/", hello)
mux.HandleFunc("/setTile", setTile)
mux.HandleFunc("/getTile", getTile)
http.ListenAndServe(":80", mux)
}
Is there something in there that needs closing? Or is it structured wrong in some way?
There seems to be lots of places to set the open file limits, so i'm not sure how to find out what the limits actually are. But it seems like increasing the limit isn't the problem anyway, surely something is being opened on every request and not closed.
This is not how you store and use a MongoDB connection in Go.
You have to store an mgo.Session, not an mgo.Database instance. And whenever you need to interact with the MongoDB, you acquire a copy or a clone of the session (e.g. with Session.Copy() or Session.Clone()), and you close it when you don't need it (preferable using a defer statement). This will ensure you don't leak connections.
You also religiously omit checking for errors, please don't do that. Whatever returns an error, do check it and act on it properly (the least you can do is print / log it).
So basically what you need to do is something like this:
var session *mgo.Session
func init() {
var err error
if session, err = mgo.Dial("localhost"); err != nil {
log.Fatal(err)
}
}
func someHandler(w http.ResponseWriter, r *http.Request) {
sess := session.Copy()
defer sess.Close() // Must close!
c := sess.DB("mapdb").C("tiles")
// Do something with the collection, e.g.
var tile bson.M
if err := c.FindId("someTileID").One(&result); err != nil {
// Tile does not exist, send back error, e.g.:
log.Printf("Tile with ID not found: %v, err: %v", "someTileID", err)
http.NotFound(w, r)
return
}
// Do something with tile
}
See related questions:
mgo - query performance seems consistently slow (500-650ms)
Concurrency in gopkg.in/mgo.v2 (Mongo, Go)
You are missing:
defer r.Body.Close()
Make sure it is used before return statement.
I'm pretty news in Golang and only use sockets in this langage for 2days. However, I'm not sure to understand something. I know in C, I used select() to know who wrote etc, but here, no one is writing until one send a message. After this message sent, my dialTCP uses it endlessly.
I think I missunderstood something about close() but I'm not sure it comes from here.. there is my code:
package dial
import (
"errors"
"encoding/json"
m "models"
"net"
"net/http"
"time"
"server"
)
type DialTCP struct {}
// tcpKeepAliveListener sets TCP keep-alive timeouts on accepted
// connections. It's used by ListenAndServe and ListenAndServeTLS so
// dead TCP connections (e.g. closing laptop mid-download) eventually
// go away.
type tcpKeepAliveListener struct {
*net.TCPListener
}
func (ln tcpKeepAliveListener) Accept() (c net.Conn, err error) {
tc, err := ln.AcceptTCP()
if err != nil {
return
}
tc.SetKeepAlive(true)
tc.SetKeepAlivePeriod(3 * time.Minute)
return tc, nil
}
func (dialTCP *DialTCP) ListenAndServe(addr string) error {
ln, err := net.Listen("tcp", addr)
if err != nil {
return err
}
return dialTCP.Serve(tcpKeepAliveListener{ln.(*net.TCPListener)})
}
func (dialTCP *DialTCP) Serve(l net.Listener) error {
defer l.Close()
for {
conn, e := l.Accept()
if e != nil {
return e
}
// you want to create server_conn here with buffers, channels and stuff
// to use async thread safe read/write from it
go dialTCP.serve_conn(conn)
}
}
func (dialTCP *DialTCP) serve_conn(conn net.Conn) error {
// var buf [512]byte
dec := json.NewDecoder(conn)
//read 1st message he sent, should be token to connect
var auth m.DialAuthentication
dec.Decode(&auth)
user := m.User{
UUID: auth.UUID,
}
ok, sb := server.IsConnected(user)
if ok == false {
json.NewEncoder(conn).Encode(sb)
return errors.New("User isn't connected.")
} else {
user.Conn = conn
}
//defer conn.Close()
var message m.DialMessageContainer
for {
dec.Decode(&message)
switch message.Type {
case ".....":
/* ....(message, user)
case "....":
....(message, user)
// case "...":*/
default:
json.NewEncoder(conn).Encode(m.StatusBack{Description: "Bad entry.", StatusId: http.StatusNotAcceptable})
}
//defer conn.Close()
}
}
I think everything is good before serv_conn(), but the error should comes from inside the for. I tried lot of things, but this for{} in Golang... Why does it have not any params/var such as C/C++/C#/Java?
for (int i = 0; i < 10; i++) {}
I'm lost about the closing of the Conn as well, so I continue to read tutorial, post and doc about how to use it.. Days pass whitout find anything
Let's say we try to communicate with a server (XMPP) which sends back XML data. We can use
conn, err := net.Dial("tcp", s.Addr+":5222")
//...
r := bufio.NewReader(conn)
//...
s, err := s.R.ReadString(10) // to read a string
But there is one problem that the server doesn't send the \10 (newline) symbol. I also tried 12 but without any luck. Same goes for readLine function as it also relies on \10. So how do I read the data sent by server?
I tried using '>' as a delimiter and succeeded to receive only parts of the messages (predictable). I had an idea to loop while error is nil and use delimiter of '>' but it also didn't work.
My research shown that the last symbol of the message is really '>' (62) and there is not any anything else at the end.
Use an xml.Decoder to read stanzas from an XMPP stream.
conn, err := net.Dial("tcp", s.Addr+":5222")
if err != nil {
// handle error
}
dec := xml.NewDecoder(conn)
Use the decoder Token method to read the root document element and to skip over character data between stanzas:
func readStartElement(dec *xml.Decoder) (xml.StartElement, error) {
for {
t, err := dec.Token()
if err != nil {
return xml.StartElement{}, err
}
switch t := t.(type) {
case xml.StartElement:
return t, nil
}
}
}
Use the decoder DecodeElement method to read a stanza:
func readStanza(dec *xml.Decoder) (interface{}, error) {
se, err := readStartElement(dec)
if err != nil {
return nil, err
}
var v interface{}
switch se.Name.Space + " " + se.Name.Local {
case "jabber:client message":
v = &jabberMessage{} // jabberMessage is struct type defined by app for messages
// Add other stanza types here.
default:
v = &struct{}{}
}
if err := dec.DecodeElement(v, &se); err != nil {
return nil, err
}
return v, nil
}
Type switch on the return value from readStanza to handle the different types of received stanzas.
A client reads stanzas synchronously. Here's rough outline (ignoring authentication, etc).
conn, err := net.Dial("tcp", s.Addr+":5222")
if err != nil {
// handle error
}
dec := xml.NewDecoder(conn)
// read and discard root element
_, err := readStartElement(dec)
if err != nil {
// handle error
}
// read stanzas
for {
v, err := readStanza(dec)
if err != nil {
// handle error
// must break out of loop on error
}
switch v := v.(type) {
case *jabberMessage:
// handle message
case *someOtherStanzaType:
// handle other stanza types
// ... and so on
}
}