So I'm having some trouble figuring out best practices for using concurrency with a MongoDB in go. My first implementation of getting a session looked like this:
var globalSession *mgo.Session
func getSession() (*mgo.Session, error) {
//Establish our database connection
if globalSession == nil {
var err error
globalSession, err = mgo.Dial(":27017")
if err != nil {
return nil, err
}
//Optional. Switch the session to a monotonic behavior.
globalSession.SetMode(mgo.Monotonic, true)
}
return globalSession.Copy(), nil
}
This works great the trouble I'm running into is that mongo has a limit of 204 connections then it starts refusing connections connection refused because too many open connections: 204;however, the issue is since I'm calling session.Copy() it only returns a session and not an error. So event though the connection refused my program never thrown an error.
Now what I though about doing is just having one session and using that instead of copy so I can have access to a connection error like so:
var session *mgo.Session = nil
func NewSession() (*mgo.Session, error) {
if session == nil {
session, err = mgo.Dial(url)
if err != nil {
return nil, err
}
}
return session, nil
}
Now the problem I have with this is that I don't know what would happen if I try to make concurrent usage of that same session.
The key is to duplicate the session and then close it when you've finished with it.
func GetMyData() []myMongoDoc {
sessionCopy, _ := getSession() // from the question above
defer sessionCopy.Close() // this is the important bit
results := make([]myMongoDoc, 0)
sessionCopy.DB("myDB").C("myCollection").Find(nil).All(&results)
return results
}
Having said that it looks like mgo doesn't actually expose control over the underlying connections (see the comment from Gustavo Niemeyer who maintains the library). A session pretty much equates to a connection, but even if you call Close() on a session mgo keeps the connection alive. From reading around it seems that Clone() might be the way to go, as it reuses the underlying socket, this will avoid the 3 way handshake of creating a new socket (see here for more discussion on the difference).
Also see this SO answer describing a standard pattern to handle sessions.
Related
I have a set of functions in my web API app. They perform some operations on the data in the Postgres database.
func CreateUser () {
db, err := sql.Open("postgres", "user=postgres password=password dbname=api_dev sslmode=disable")
// Do some db operations here
}
I suppose functions should work with db independently from each other, so now I have sql.Open(...) inside each function. I don't know if it's a correct way to manage db connection.
Should I open it somewhere once the app starts and pass db as an argument to the corresponding functions instead of opening the connection in every function?
Opening a db connection every time it's needed is a waste of resources and it's slow.
Instead, you should create an sql.DB once, when your application starts (or on first demand), and either pass it where it is needed (e.g. as a function parameter or via some context), or simply make it a global variable and so everyone can access it. It's safe to call from multiple goroutines.
Quoting from the doc of sql.Open():
The returned DB is safe for concurrent use by multiple goroutines and maintains its own pool of idle connections. Thus, the Open function should be called just once. It is rarely necessary to close a DB.
You may use a package init() function to initialize it:
var db *sql.DB
func init() {
var err error
db, err = sql.Open("yourdriver", "yourDs")
if err != nil {
log.Fatal("Invalid DB config:", err)
}
}
One thing to note here is that sql.Open() may not create an actual connection to your DB, it may just validate its arguments. To test if you can actually connect to the db, use DB.Ping(), e.g.:
func init() {
var err error
db, err = sql.Open("yourdriver", "yourDs")
if err != nil {
log.Fatal("Invalid DB config:", err)
}
if err = db.Ping(); err != nil {
log.Fatal("DB unreachable:", err)
}
}
I will use a postgres example
package main
import necessary packages and don't forget the postgres driver
import (
"database/sql"
_ "github.com/lib/pq" //postgres driver
)
initialize your connection in the package scope
var db *sql.DB
have an init function for your connection
func init() {
var err error
db, err = sql.open("postgres", "connectionString")
//connectioString example => 'postgres://username:password#localhost/dbName?sslmode=disable'
if err != nil {
panic(err)
}
err = db.Ping()
if err != nil {
panic(err)
}
// note, we haven't deffered db.Close() at the init function since the connection will close after init. you could close it at main or ommit it
}
main function
func main() {
defer db.Close() //optional
//run your db functions
}
checkout this example
https://play.golang.org/p/FAiGbqeJG0H
As per the documentation in Readme:
Make sure to defer a call to Disconnect after instantiating your client:
defer func() {
if err = client.Disconnect(ctx); err != nil {
panic(err)
}
}()
Does the above documentation meant to disconnect, during shutdown of the program(using mongodb driver)?
They just reminded you that you should always close the connection to the database at some point. When exactly is up to you. Usually, you initialize the database connection at the top-level of your application, so the defer call should be at the same level. For example,
func main() {
client, err := mongo.Connect(ctx, options.Client().ApplyURI("mongodb://localhost:27017"))
defer func() {
if err = client.Disconnect(ctx); err != nil {
panic(err)
}
}()
// Pass the connection to other components of your appliaction
doSomeWork(client)
}
Note: If you are using Goroutines don't forget to synchronize them in main otherwise the connection will be close too early.
I think the documentation meant to use defer with client.Disconnect in the main function of your program. Thanks to that your program will close all the MongoDB client connections before exiting.
If you would use it e.g. in a helper function that prepares that client, it would close all the connections right after the client creation which may not be something you want.
How do I create a proper mongo based application in Go using the official driver(go.mongodb.org/mongo-driver/mongo)? I have a MongoConnect() and MongoDisconnect(client) function to create a connection and delete it. But, it's not too efficient and starts leaking FD as the app has got around 40 functions and finding all the missed MongoDisconnect() becomes hectic.
The current MongoConnect and MongoDisconnect are as follows.
func MongoConnect() (*mongo.Client, error) {
clientOptions := options.Client().ApplyURI("mongodb://localhost:27017")
client, err := mongo.Connect(context.TODO(), clientOptions)
if err != nil {
Logger(err)
return nil, err
}
err = client.Ping(context.TODO(), nil)
if err != nil {
Logger(err)
return nil, err
}
return client, err
}
func MongoDisconnect(client *mongo.Client) {
_ = client.Disconnect(context.TODO())
}
I am looking for a method that would still use MongoConnect() to create connections and would automatically kill the client without the usage of MongoDisconnect().
PS. Other methods that are better than the above requirement are also welcome
I'm not sure that there is an 'efficient' way to fix the underlying issue, you probably will need to look at all the places where you've called MongoConnect() and ensure you have a corresponding MongoDisconnect().
In saying that, what you might want to look at is implemententing the right pattern for connecting to databases.
Generally speaking, if your database driver takes care of managing connections for you then you should create the connection once and just pass it around as needed.
You could also defer the closing of that connection to a go routine which would close it once it was no longer needed (when you're application is shutting down).
Here is a code snippet of how this is implemented:
// =========================================================================
// Start Database
log.Println("main: Initializing database support")
db, err := database.Open(database.Config{
User: cfg.DB.User,
Password: cfg.DB.Password,
Host: cfg.DB.Host,
Name: cfg.DB.Name,
DisableTLS: cfg.DB.DisableTLS,
})
if err != nil {
return errors.Wrap(err, "connecting to db")
}
defer func() {
log.Printf("main: Database Stopping : %s", cfg.DB.Host)
db.Close()
}()
I didn't write this code and its part of a larger project scaffold which has some other nice patterns for web applications.
Ultimate service
I'm using MongoDB (gopkg.in/mgo.v2 package) as a database in my go app. According to MongoDB best practices I should to open connection when application starting and close it when application is terminating. To verify that connection will be closed I can use defer construction:
session, err := mgo.Dial(mongodbURL)
if err != nil {
panic(err)
}
defer session.Close()
All will be good if I execute this code in main function. But I want to have this code in separate go file. If I do this session will be closed after method will be executed.What is the best way to open and close session in Golang according MongoDB best practices?
You can do something like this. Create a package which does the Db initialization
package common
import "gopkg.in/mgo.v2"
var mgoSession *mgo.Session
// Creates a new session if mgoSession is nil i.e there is no active mongo session.
//If there is an active mongo session it will return a Clone
func GetMongoSession() *mgo.Session {
if mgoSession == nil {
var err error
mgoSession, err = mgo.Dial(mongo_conn_str)
if err != nil {
log.Fatal("Failed to start the Mongo session")
}
}
return mgoSession.Clone()
}
Clone reuses the same socket as the original session.
Now in other packages you can call this method:
package main
session := common.GetMongoSession()
defer session.Close()
Pass the section to the other part of the code
after the defer(),
func main(){
// ... other stuff
session, err := mgo.Dial(mongodbURL)
if err != nil {
panic(err)
}
defer session.Close()
doThinginOtherFile(session)
}
It looks like you can clone/copy sessions if necessary as long as you have one to clone from.
So I'm trying to use unix sockets with fluentd for a logging task and find that randomly, once in a while the error
dial: {socket_name} resource temporarily unavailable
Any ideas as to why this might be occurring?
I tried adding "retry" logic, to reduce the error, but it still occurs at times.
Also, for fluntd we are using the default config for unix sockets communication
func connect() {
var connection net.Conn
var err error
for i := 0; i < retry_count; i++ {
connection, err = net.Dial("unix", path_to_socket)
if err == nil {
break
}
time.Sleep(time.Duration(math.Exp2(float64(retry_count))) * time.Millisecond)
}
if err != nil {
fmt.Println(err)
} else {
connection.Write(data_to_send_socket)
}
defer connection.Close()
}
Go creates its sockets in non-blocking mode, which means that certain system calls that would usually block instead. In most cases it transparently handles the EAGAIN error (what is indicated by the "resource temporarily unavailable" message) by waiting until the socket is ready to read/write. It doesn't seem to have this logic for the connect call in Dial though.
It is possible for connect to return EAGAIN when connecting to a UNIX domain socket if its listen queue has filled up. This will happen if clients are connecting to it faster than it is accepting them. Go should probably wait on the socket until it becomes connectable in this case and retry similar to what it does for Read/Write, but it doesn't seem to have that logic.
So your best bet would be to handle the error by waiting and retrying the Dial call. That, or work out why your server isn't accepting connections in a timely manner.
For the exponential backoff you can use this library: github.com/cenkalti/backoff. I think the way you have it now it always sleeps for the same amount of time.
For the network error you need to check if it's a temporary error or not. If it is then retry:
type TemporaryError interface {
Temporary() bool
}
func dial() (conn net.Conn, err error) {
backoff.Retry(func() error {
conn, err = net.Dial("unix", "/tmp/ex.socket")
if err != nil {
// if this is a temporary error, then retry
if terr, ok := err.(TemporaryError); ok && terr.Temporary() {
return err
}
}
// if we were successful, or there was a non-temporary error, fail
return nil
}, backoff.NewExponentialBackOff())
return
}