As per the documentation in Readme:
Make sure to defer a call to Disconnect after instantiating your client:
defer func() {
if err = client.Disconnect(ctx); err != nil {
panic(err)
}
}()
Does the above documentation meant to disconnect, during shutdown of the program(using mongodb driver)?
They just reminded you that you should always close the connection to the database at some point. When exactly is up to you. Usually, you initialize the database connection at the top-level of your application, so the defer call should be at the same level. For example,
func main() {
client, err := mongo.Connect(ctx, options.Client().ApplyURI("mongodb://localhost:27017"))
defer func() {
if err = client.Disconnect(ctx); err != nil {
panic(err)
}
}()
// Pass the connection to other components of your appliaction
doSomeWork(client)
}
Note: If you are using Goroutines don't forget to synchronize them in main otherwise the connection will be close too early.
I think the documentation meant to use defer with client.Disconnect in the main function of your program. Thanks to that your program will close all the MongoDB client connections before exiting.
If you would use it e.g. in a helper function that prepares that client, it would close all the connections right after the client creation which may not be something you want.
Related
I have a set of functions in my web API app. They perform some operations on the data in the Postgres database.
func CreateUser () {
db, err := sql.Open("postgres", "user=postgres password=password dbname=api_dev sslmode=disable")
// Do some db operations here
}
I suppose functions should work with db independently from each other, so now I have sql.Open(...) inside each function. I don't know if it's a correct way to manage db connection.
Should I open it somewhere once the app starts and pass db as an argument to the corresponding functions instead of opening the connection in every function?
Opening a db connection every time it's needed is a waste of resources and it's slow.
Instead, you should create an sql.DB once, when your application starts (or on first demand), and either pass it where it is needed (e.g. as a function parameter or via some context), or simply make it a global variable and so everyone can access it. It's safe to call from multiple goroutines.
Quoting from the doc of sql.Open():
The returned DB is safe for concurrent use by multiple goroutines and maintains its own pool of idle connections. Thus, the Open function should be called just once. It is rarely necessary to close a DB.
You may use a package init() function to initialize it:
var db *sql.DB
func init() {
var err error
db, err = sql.Open("yourdriver", "yourDs")
if err != nil {
log.Fatal("Invalid DB config:", err)
}
}
One thing to note here is that sql.Open() may not create an actual connection to your DB, it may just validate its arguments. To test if you can actually connect to the db, use DB.Ping(), e.g.:
func init() {
var err error
db, err = sql.Open("yourdriver", "yourDs")
if err != nil {
log.Fatal("Invalid DB config:", err)
}
if err = db.Ping(); err != nil {
log.Fatal("DB unreachable:", err)
}
}
I will use a postgres example
package main
import necessary packages and don't forget the postgres driver
import (
"database/sql"
_ "github.com/lib/pq" //postgres driver
)
initialize your connection in the package scope
var db *sql.DB
have an init function for your connection
func init() {
var err error
db, err = sql.open("postgres", "connectionString")
//connectioString example => 'postgres://username:password#localhost/dbName?sslmode=disable'
if err != nil {
panic(err)
}
err = db.Ping()
if err != nil {
panic(err)
}
// note, we haven't deffered db.Close() at the init function since the connection will close after init. you could close it at main or ommit it
}
main function
func main() {
defer db.Close() //optional
//run your db functions
}
checkout this example
https://play.golang.org/p/FAiGbqeJG0H
How do I create a proper mongo based application in Go using the official driver(go.mongodb.org/mongo-driver/mongo)? I have a MongoConnect() and MongoDisconnect(client) function to create a connection and delete it. But, it's not too efficient and starts leaking FD as the app has got around 40 functions and finding all the missed MongoDisconnect() becomes hectic.
The current MongoConnect and MongoDisconnect are as follows.
func MongoConnect() (*mongo.Client, error) {
clientOptions := options.Client().ApplyURI("mongodb://localhost:27017")
client, err := mongo.Connect(context.TODO(), clientOptions)
if err != nil {
Logger(err)
return nil, err
}
err = client.Ping(context.TODO(), nil)
if err != nil {
Logger(err)
return nil, err
}
return client, err
}
func MongoDisconnect(client *mongo.Client) {
_ = client.Disconnect(context.TODO())
}
I am looking for a method that would still use MongoConnect() to create connections and would automatically kill the client without the usage of MongoDisconnect().
PS. Other methods that are better than the above requirement are also welcome
I'm not sure that there is an 'efficient' way to fix the underlying issue, you probably will need to look at all the places where you've called MongoConnect() and ensure you have a corresponding MongoDisconnect().
In saying that, what you might want to look at is implemententing the right pattern for connecting to databases.
Generally speaking, if your database driver takes care of managing connections for you then you should create the connection once and just pass it around as needed.
You could also defer the closing of that connection to a go routine which would close it once it was no longer needed (when you're application is shutting down).
Here is a code snippet of how this is implemented:
// =========================================================================
// Start Database
log.Println("main: Initializing database support")
db, err := database.Open(database.Config{
User: cfg.DB.User,
Password: cfg.DB.Password,
Host: cfg.DB.Host,
Name: cfg.DB.Name,
DisableTLS: cfg.DB.DisableTLS,
})
if err != nil {
return errors.Wrap(err, "connecting to db")
}
defer func() {
log.Printf("main: Database Stopping : %s", cfg.DB.Host)
db.Close()
}()
I didn't write this code and its part of a larger project scaffold which has some other nice patterns for web applications.
Ultimate service
I have a websocket client. In reality, it is far more complex than the basic code shown below.
I now need to scale this client code to open connections to multiple servers. Ultimately, the tasks that need to be performed when a message is received from the servers is identical.
What would be the best approach to handle this?
As I said above the actual code performed when receiving the message is far more complex than shown in the example.
package main
import (
"flag"
"log"
"net/url"
"os"
"os/signal"
"time"
"github.com/gorilla/websocket"
)
var addr = flag.String("addr", "localhost:1234", "http service address")
func main() {
flag.Parse()
log.SetFlags(0)
interrupt := make(chan os.Signal, 1)
signal.Notify(interrupt, os.Interrupt)
// u := url.URL{Scheme: "ws", Host: *addr, Path: "/echo"}
u := url.URL{Scheme: "ws", Host: *addr, Path: "/"}
log.Printf("connecting to %s", u.String())
c, _, err := websocket.DefaultDialer.Dial(u.String(), nil)
if err != nil {
log.Fatal("dial:", err)
}
defer c.Close()
done := make(chan struct{})
go func() {
defer close(done)
for {
_, message, err := c.ReadMessage()
if err != nil {
log.Println("read:", err)
return
}
log.Printf("recv: %s", message)
}
}()
ticker := time.NewTicker(time.Second)
defer ticker.Stop()
for {
select {
case <-done:
return
case t := <-ticker.C:
err := c.WriteMessage(websocket.TextMessage, []byte(t.String()))
if err != nil {
log.Println("write:", err)
return
}
case <-interrupt:
log.Println("interrupt")
// Cleanly close the connection by sending a close message and then
// waiting (with timeout) for the server to close the connection.
err := c.WriteMessage(websocket.CloseMessage, websocket.FormatCloseMessage(websocket.CloseNormalClosure, ""))
if err != nil {
log.Println("write close:", err)
return
}
select {
case <-done:
case <-time.After(time.Second):
}
return
}
}
}
Modify the interrupt handling to close a channel on interrupt. This allows multiple goroutines to wait on the event by waiting for the channel to close.
shutdown := make(chan struct{})
interrupt := make(chan os.Signal, 1)
signal.Notify(interrupt, os.Interrupt)
go func() {
<-interrupt
log.Println("interrupt")
close(shutdown)
}()
Move the per-connection code to a function. This code is a copy and paste from the question with two changes: the interrupt channel is replaced with the shutdown channel; the function notifies a sync.WaitGroup when the function is done.
func connect(u string, shutdown chan struct{}, wg *sync.WaitGroup) {
defer wg.Done()
log.Printf("connecting to %s", u)
c, _, err := websocket.DefaultDialer.Dial(u, nil)
if err != nil {
log.Fatal("dial:", err)
}
defer c.Close()
done := make(chan struct{})
go func() {
defer close(done)
for {
_, message, err := c.ReadMessage()
if err != nil {
log.Println("read:", err)
return
}
log.Printf("recv: %s", message)
}
}()
ticker := time.NewTicker(time.Second)
defer ticker.Stop()
for {
select {
case <-done:
return
case t := <-ticker.C:
err := c.WriteMessage(websocket.TextMessage, []byte(t.String()))
if err != nil {
log.Println("write:", err)
return
}
case <-shutdown:
// Cleanly close the connection by sending a close message and then
// waiting (with timeout) for the server to close the connection.
err := c.WriteMessage(websocket.CloseMessage, websocket.FormatCloseMessage(websocket.CloseNormalClosure, ""))
if err != nil {
log.Println("write close:", err)
return
}
select {
case <-done:
case <-time.After(time.Second):
}
return
}
}
}
Declare a sync.WaitGroup in main(). For each websocket endpoint that you want to connect to, increment the WaitGroup and start a goroutine to connect that endpoint. After starting the goroutines, wait on the WaitGroup for the goroutines to complete.
var wg sync.WaitGroup
for _, u := range endpoints { // endpoints is []string
// where elements are URLs
// of endpoints to connect to.
wg.Add(1)
go connect(u, shutdown, &wg)
}
wg.Wait()
The code above with an edit to make it run against Gorilla's echo example server is posted on the playground.
is the communication with every different server completely independendant of the other servers? if yes i would go around in a fashion like:
in main create a context with a cancellation function
create a waitgroup in main to track fired up goroutines
for every server, add to the waitgroup, fire up a new goroutine from the main function passing the context and the waitgroup references
main goes in a for/select loop listening to for signals and if one arrives calls the cancelfunc and waits on the waitgroup.
main can also listen on a result chan from the goroutines and maybe print the results itself it the goroutines shouldn't do it directly.
every goroutine has as we said has references for the wg, the context and possibly a chan to return results. now the approach splits on if the goroutine must do one and one thing only, or if it needs to do a sequence of things. for the first approach
if only one thing is to be done we follow an approach like the one descripbed here (observe that to be asyncronous he would in turn fire up a new goroutine to perform the DoSomething() step that would return the result on the channel)
That allows it to be able to accept the cancellation signal at any time. it is up to you to determine how non-blocking you want to be and how prompt you want to be to respond to cancellation signals.Also the benefit of having the a context associated being passed to the goroutines is that you can call the Context enabled versions of most library functions. For example if you want your dials to have a timeout of let's say 1 minute, you would create a new context with timeout from the one passed and then DialContext with that. This allows the dial to stop both from a timeout or the parent (the one you created in main) context's cancelfunc being called.
if more things need to be done ,i usually prefer to do one thing with the goroutine, have it invoke a new one with the next step to be performed (passing all the references down the pipeline) and exit.
this approach scales well with cancellations and being able to stop the pipeline at any step as well as support contexts with dealines easily for steps that can take too long.
I'm using MongoDB (gopkg.in/mgo.v2 package) as a database in my go app. According to MongoDB best practices I should to open connection when application starting and close it when application is terminating. To verify that connection will be closed I can use defer construction:
session, err := mgo.Dial(mongodbURL)
if err != nil {
panic(err)
}
defer session.Close()
All will be good if I execute this code in main function. But I want to have this code in separate go file. If I do this session will be closed after method will be executed.What is the best way to open and close session in Golang according MongoDB best practices?
You can do something like this. Create a package which does the Db initialization
package common
import "gopkg.in/mgo.v2"
var mgoSession *mgo.Session
// Creates a new session if mgoSession is nil i.e there is no active mongo session.
//If there is an active mongo session it will return a Clone
func GetMongoSession() *mgo.Session {
if mgoSession == nil {
var err error
mgoSession, err = mgo.Dial(mongo_conn_str)
if err != nil {
log.Fatal("Failed to start the Mongo session")
}
}
return mgoSession.Clone()
}
Clone reuses the same socket as the original session.
Now in other packages you can call this method:
package main
session := common.GetMongoSession()
defer session.Close()
Pass the section to the other part of the code
after the defer(),
func main(){
// ... other stuff
session, err := mgo.Dial(mongodbURL)
if err != nil {
panic(err)
}
defer session.Close()
doThinginOtherFile(session)
}
It looks like you can clone/copy sessions if necessary as long as you have one to clone from.
When I send requests from the following code:
req, err := http.NewRequest("GET", "my_target:", nil)
if err != nil {
panic(err)
}
req.Close = true
client := http.DefaultClient
resp, err := client.Do(req)
if err != nil {
panic(err)
}
defer resp.Body.Close()
After a few hours of sending average 10 requests per minutes, I get this error:
socket: too many open files
How do I find the cause of this problem?
Am I not closing the http.Request?
I thought req.Close = true does that job.
Thanks!
Why are you deferring the close? Are you actually reading from this body?
defer resp.Body.Close()
Do you actually return from the current function before performing another Get? If not, then the defer will never execute, and you'll never release this connection for reuse.
req.Close = true is an unusual choice here, as well. This also prevents connection reuse, which is something you'd probably want rather than forbid. This doesn't automatically close the request on your side. It forces the server to immediately close the connection, which you would otherwise reuse. You'll hold your side open until you close it.
Typically for a simple GET request like you have here, I would just do this:
resp, err := http.Get("...")
if err != nil {
panic(err) // panic seems harsh, usually you'd just return the error. But either way...
}
resp.Body.Close()
There's no need for a special client here. Just use the default one. It'll take care of the rest as long as you make sure to close the response body. And there's no need to complicate things with a defer if you're going to immediately close the body. The reason for defer is to make sure you close the body if you have a bunch of processing later that might return or panic on error.