Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 1 year ago.
Improve this question
My question is that should I connect to MongoDB in all of my handlers using goroutines and then disconnect the connection.
Or I should just connect to MongoDB when app starts and keep connection alive for a long time and use that connection in my handlers.
What is the best approach?
I would be thankful if you explain the advantages and disadvantages.
The latter is better
Connect to MongoDB when app starts and keep connection alive for a long time and use that connection in my handlers
it prevents you from having to connect to the database when you need to interact with the database all the time and having to deal with cases where connection to the database is inconsistent, the former might lead to a lot of complexity.
Conventionally, connecting to your db should occur once (probably in your main.go file) and you can reference the connection in other parts of the project.
My approach that has proven to be quite performant.
Define an internal data package like so :
package data
import (
"context"
"log"
"os"
"time"
"github.com/joho/godotenv"
"go.mongodb.org/mongo-driver/mongo"
"go.mongodb.org/mongo-driver/mongo/options"
)
func MongoConnect() (*mongo.Client, error) {
godotenv.Load(".env")
str := os.Getenv("CONNECTION_STRING")
client, err := mongo.NewClient(options.Client().ApplyURI(str))
if err != nil {
log.Fatal(err)
}
ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
defer cancel()
err = client.Connect(ctx)
if err != nil {
log.Fatal(err)
}
return client, nil
}
func ConnectToCollection(collectionName string) (*mongo.Collection, error) {
client, err := MongoConnect()
if err != nil {
return nil, err
}
collection := client.Database("DATABASE_NAME").Collection(collectionName)
return collection, nil
}
Once this is done, you can basically use this ConnectToCollection in every endpoint group (package) that needs it in a global variable. E.G:
package xxx
import "app_name/whatever/data"
var coll, _ = data.ConnectToCollection("xxx_collection_name")
func myFunction() {
coll.InsertOne(...)
}
Related
How do I create a proper mongo based application in Go using the official driver(go.mongodb.org/mongo-driver/mongo)? I have a MongoConnect() and MongoDisconnect(client) function to create a connection and delete it. But, it's not too efficient and starts leaking FD as the app has got around 40 functions and finding all the missed MongoDisconnect() becomes hectic.
The current MongoConnect and MongoDisconnect are as follows.
func MongoConnect() (*mongo.Client, error) {
clientOptions := options.Client().ApplyURI("mongodb://localhost:27017")
client, err := mongo.Connect(context.TODO(), clientOptions)
if err != nil {
Logger(err)
return nil, err
}
err = client.Ping(context.TODO(), nil)
if err != nil {
Logger(err)
return nil, err
}
return client, err
}
func MongoDisconnect(client *mongo.Client) {
_ = client.Disconnect(context.TODO())
}
I am looking for a method that would still use MongoConnect() to create connections and would automatically kill the client without the usage of MongoDisconnect().
PS. Other methods that are better than the above requirement are also welcome
I'm not sure that there is an 'efficient' way to fix the underlying issue, you probably will need to look at all the places where you've called MongoConnect() and ensure you have a corresponding MongoDisconnect().
In saying that, what you might want to look at is implemententing the right pattern for connecting to databases.
Generally speaking, if your database driver takes care of managing connections for you then you should create the connection once and just pass it around as needed.
You could also defer the closing of that connection to a go routine which would close it once it was no longer needed (when you're application is shutting down).
Here is a code snippet of how this is implemented:
// =========================================================================
// Start Database
log.Println("main: Initializing database support")
db, err := database.Open(database.Config{
User: cfg.DB.User,
Password: cfg.DB.Password,
Host: cfg.DB.Host,
Name: cfg.DB.Name,
DisableTLS: cfg.DB.DisableTLS,
})
if err != nil {
return errors.Wrap(err, "connecting to db")
}
defer func() {
log.Printf("main: Database Stopping : %s", cfg.DB.Host)
db.Close()
}()
I didn't write this code and its part of a larger project scaffold which has some other nice patterns for web applications.
Ultimate service
This question already has answers here:
golang what is import side effect
(1 answer)
What does an underscore in front of an import statement mean?
(5 answers)
Import side effects
(1 answer)
Closed 4 years ago.
I continue to get this issue and I do not understand why
package models
import (
"database/sql"
"fmt"
_ "github.com/lib/pq"
)
const (
host = "localhost"
port = 5432
user = "postgres"
password = "postgres"
dbname = "postgres"
)
var db *sql.DB
func InitDB() {
psqlInfo := fmt.Sprintf("host=%s port=%d user=%s "+
"password=%s dbname=%s sslmode=disable",
host, port, user, password, dbname)
db, err := sql.Open("postgres", psqlInfo)
if err != nil {
panic(err)
}
defer db.Close()
err = db.Ping()
if err != nil {
panic(err)
}
fmt.Println("Successfully connected!")
}
Apologies as I am new to go and attemping to understand it. Thank you for any help. I have attempted to pull the library as well as move it around
Assuming you're referring to
import (
_ "github.com/lib/pq"
)
This is importing for a side effect. The underscore is in the alias position. This allows you to circumvent the error (or automatic removal if using something like goimports) of an unused import.
When github.com/lib/pq is imported it runs an init() function that registers the postgres database driver.
This init function can be seen here https://github.com/lib/pq/blob/d34b9ff171c21ad295489235aec8b6626023cd04/conn.go#L48-L50
func init() {
sql.Register("postgres", &Driver{})
}
If you needed to refer to something in the pq package itself it is fine to remove the underscore and then make direct reference to the package. However, make sure you do not unintentionally remove the import entirely or your attempt to open a connection to the postgres database will fail.
Closed. This question needs debugging details. It is not currently accepting answers.
Edit the question to include desired behavior, a specific problem or error, and the shortest code necessary to reproduce the problem. This will help others answer the question.
Closed 1 year ago.
Improve this question
I want to batch insert some rows and get their ids with golang, this is my attempt in doing so
import (
"database/sql"
"fmt"
_ "github.com/bmizerany/pq"
"log"
)
func main() {
conn := fmt.Sprintf("host=%s user=%s password=%s dbname=%s sslmode=require", host, user, password, dbname)
d, err := sql.Open("postgres", conn)
rows, err := d.Query("INSERT INTO MYTABLE(MYCOLUMN) VALUES(1),(2),(3) RETURNING ID")
defer rows.Close()
var ids []int
for rows.Next() {
var id int
scan_err := rows.Scan(&id)
fmt.Println(scan_err)
if scan_err != nil {
log.Fatal(scan_err)
}
ids = append(ids, id)
fmt.Printf("id %d\n", id)
}
}
my problem is ids is always empty even if the values are inserted correctly in the database.
what's interesting is that the following works
var id int
db.QueryRow("INSERT INTO MYTABLE(MYCOLUMN) VALUES(1) RETURNING ID").Scan(&id)
How do solve this?
I wonder about MongoDB session management in Go using mgo, especially about how to correctly ensure a session is closed and how to react on write failures.
I have read the following:
Best practice to maintain a mgo session
Should I copy session for each operation in mgo?
Still, cannot apply it to my situation.
I have two goroutines which store event after event into MongoDB sharing the same *mgo.Session, both looking essiantially like the following:
func storeEvents(session *mgo.Session) {
session_copy := session.Copy()
// *** is it correct to defer the session close here? <-----
defer session_copy.Close()
col := session_copy.DB("DB_NAME").C("COLLECTION_NAME")
for {
event := GetEvent()
err := col.Insert(&event)
if err != nil {
// *** insert FAILED - how to react properly? <-----
session_copy = session.Copy()
defer session_copy.Close()
}
}
}
col.Insert(&event) after some hours returns the error
read tcp 127.0.0.1:46954->127.0.0.1:27017: i/o timeout
and I am unsure how to react on this. After this error occurs, it occurs on all subsequent writes, hence it seems I have to create a new session. Alternatives for me seem:
1) restart the whole goroutine, i.e.
if err != nil {
go storeEvents(session)
return
}
2) create a new session copy
if err != nil {
session_copy = session.Copy()
defer session_copy.Close()
col := session_copy.DB("DB_NAME").C("COLLECTION_NAME")
continue
}
--> Is it correct how I use defer session_copy.Close()? (Note the above defer references the Close() function of another session. Anyway, those sessions will never be closed since the function never returns. I.e., with time, many sessions will be created and not closed.
Other options?
So I don't know if this is going to help you any, but I don't have any issues with this set up.
I have a mongo package that I import from. This is a template of my mongo.go file
package mongo
import (
"time"
"gopkg.in/mgo.v2"
)
var (
// MyDB ...
MyDB DataStore
)
// create the session before main starts
func init() {
MyDB.ConnectToDB()
}
// DataStore containing a pointer to a mgo session
type DataStore struct {
Session *mgo.Session
}
// ConnectToTagserver is a helper method that connections to pubgears' tagserver
// database
func (ds *DataStore) ConnectToDB() {
mongoDBDialInfo := &mgo.DialInfo{
Addrs: []string{"ip"},
Timeout: 60 * time.Second,
Database: "db",
}
sess, err := mgo.DialWithInfo(mongoDBDialInfo)
if err != nil {
panic(err)
}
sess.SetMode(mgo.Monotonic, true)
MyDB.Session = sess
}
// Close is a helper method that ensures the session is properly terminated
func (ds *DataStore) Close() {
ds.Session.Close()
}
Then in another package, for example main Updated Based on the comment below
package main
import (
"../models/mongo"
)
func main() {
// Grab the main session which was instantiated in the mongo package init function
sess := mongo.MyDB.Session
// pass that session in
storeEvents(sess)
}
func storeEvents(session *mgo.Session) {
session_copy := session.Copy()
defer session_copy.Close()
// Handle panics in a deferred fuction
// You can turn this into a wrapper (middleware)
// remove this this function, and just wrap your calls with it, using switch cases
// you can handle all types of errors
defer func(session *mgo.Session) {
if err := recover(); err != nil {
fmt.Printf("Mongo insert has caused a panic: %s\n", err)
fmt.Println("Attempting to insert again")
session_copy := session.Copy()
defer session_copy.Close()
col := session_copy.DB("DB_NAME").C("COLLECTION_NAME")
event := GetEvent()
err := col.Insert(&event)
if err != nil {
fmt.Println("Attempting to insert again failed")
return
}
fmt.Println("Attempting to insert again succesful")
}
}(session)
col := session_copy.DB("DB_NAME").C("COLLECTION_NAME")
event := GetEvent()
err := col.Insert(&event)
if err != nil {
panic(err)
}
}
I use a similar setup on my production servers on AWS. I do over 1 million inserts an hour. Hope this helps. Another things I've done to ensure that the mongo servers can handle the connections is increate the ulimit on my production machines. It's talked about in this stack
I'm using MongoDB (gopkg.in/mgo.v2 package) as a database in my go app. According to MongoDB best practices I should to open connection when application starting and close it when application is terminating. To verify that connection will be closed I can use defer construction:
session, err := mgo.Dial(mongodbURL)
if err != nil {
panic(err)
}
defer session.Close()
All will be good if I execute this code in main function. But I want to have this code in separate go file. If I do this session will be closed after method will be executed.What is the best way to open and close session in Golang according MongoDB best practices?
You can do something like this. Create a package which does the Db initialization
package common
import "gopkg.in/mgo.v2"
var mgoSession *mgo.Session
// Creates a new session if mgoSession is nil i.e there is no active mongo session.
//If there is an active mongo session it will return a Clone
func GetMongoSession() *mgo.Session {
if mgoSession == nil {
var err error
mgoSession, err = mgo.Dial(mongo_conn_str)
if err != nil {
log.Fatal("Failed to start the Mongo session")
}
}
return mgoSession.Clone()
}
Clone reuses the same socket as the original session.
Now in other packages you can call this method:
package main
session := common.GetMongoSession()
defer session.Close()
Pass the section to the other part of the code
after the defer(),
func main(){
// ... other stuff
session, err := mgo.Dial(mongodbURL)
if err != nil {
panic(err)
}
defer session.Close()
doThinginOtherFile(session)
}
It looks like you can clone/copy sessions if necessary as long as you have one to clone from.