How to Manage MongoDB Client across Packages in Golang - mongodb

I am currently starting out with MongoDB on GoLang. My current use case is this.
I wish to initialize connection to my MongoDB Database in a particular package and use the returned client across several other local packages. Here is what I have tried,
I have initialized the connection to MongoDB inside a package called dataLayer as below
package dataLayer
import (
"context"
"log"
"time"
"go.mongodb.org/mongo-driver/mongo"
"go.mongodb.org/mongo-driver/mongo/options"
"go.mongodb.org/mongo-driver/mongo/readpref"
)
func InitDataLayer() {
ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
defer cancel()
client, err := mongo.Connect(ctx, options.Client().ApplyURI(credentials.MONGO_DB_ATLAS_URI))
if err != nil {
log.Fatal(err)
} else {
log.Println("Connected to Database")
}
}
Now if I wish to use the returned client in other packages, is it production safe to keep on calling initDataLayer over and over again to get back a client?
Thanks alot.

You don't have to call InitDataLayer over and over again. You need to create only one client, you can use the same client to connect to Mongo DB from various places.
Mongo client internally maintains a pool of connections, so you don't have to create this client again and again.
It's a good design to store this client as the struct field and keep reusing the same.
Edit:
Create Connection
package dataLayer
import (
"context"
"log"
"time"
"go.mongodb.org/mongo-driver/mongo"
"go.mongodb.org/mongo-driver/mongo/options"
"go.mongodb.org/mongo-driver/mongo/readpref"
)
func InitDataLayer()(*mongo.Client) {
ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
defer cancel()
client, err := mongo.Connect(ctx, options.Client().ApplyURI(credentials.MONGO_DB_ATLAS_URI))
if err != nil {
log.Fatal(err)
} else {
log.Println("Connected to Database")
}
return client
}
main.go,
Close connection at the end of usage, otherwise connection would be leaked.
func main(){
//...
client = dataLayer.InitDataLayer()
defer client.Disconnect(context.Background())
...
}
Repository/DAL, store client in Repository struct
Interface
type BookRepository interface {
FindById( ctx context.Context, id int) (*Book, error)
}
Repository Implementation that store client in struct
type bookRepository struct {
client *mongo.Client
}
func (r *bookRepository) FindById( ctx context.Context, id int) (*Book, error) {
var book Book
err := r.client.DefaultDatabase().Collection("books").FindOne(ctx, bson.M{"_id": id }).Decode(&book)
if err == mongo.ErrNoDocuments {
return nil, nil
}
if err != nil {
return nil, err
}
return &book, nil
}

Related

Can the MongoDB client be reused? [duplicate]

This question already has answers here:
Is mongodb client driver concurrent safe?
(1 answer)
goroutine create multiple mongodb connection
(1 answer)
Closed 21 days ago.
I've managed to connect to MongoDB and I've got this client object, can this be passed around and reused?
Or does it need to be created each time I want to interact with the database?
If it can be reused then ill move the database connection out to its own function and run it just once at startup.
package main
import (
"context"
"fmt"
"log"
"main/api"
"os"
"time"
"github.com/joho/godotenv"
"go.mongodb.org/mongo-driver/mongo"
"go.mongodb.org/mongo-driver/mongo/options"
)
type User struct {
Username string
Password string `bson:"password,omitempty"`
}
func main() {
if err := godotenv.Load(); err != nil {
log.Println("No .env file found")
}
uri := os.Getenv("MONGODB_URI")
serverAPIOptions := options.ServerAPI(options.ServerAPIVersion1)
clientOptions := options.Client().
ApplyURI(uri).
SetServerAPIOptions(serverAPIOptions)
ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
defer cancel()
client, err := mongo.Connect(ctx, clientOptions)
if err != nil {
log.Fatal(err)
}
usersColl := client.Database("sampleShop").Collection("users")
newUser := User{Username: "8282", Password: "Korean"}
result, err := usersColl.InsertOne(context.TODO(), newUser)
if err != nil {
panic(err)
}
fmt.Println(result)
fmt.Println(uri)
api.GetPaymentSession()
}

Mongo client set in main function, functions in other modules receive nil value

I have a restful API utilizing mux and mongo-driver. Following a tutorial, I attempted to setup the server and mongo client like so in the main package:
package main
import (
"context"
"fmt"
"net/http"
"time"
"github.com/gorilla/mux"
c "github.com/moonlightfight/elo-backend/config"
"github.com/moonlightfight/elo-backend/routes/admin"
"github.com/moonlightfight/elo-backend/routes/tournament"
"github.com/spf13/viper"
"go.mongodb.org/mongo-driver/mongo"
"go.mongodb.org/mongo-driver/mongo/options"
)
var client *mongo.Client
func main() {
// Set the file name of the configurations file
viper.SetConfigName("config")
// Set the path to look for the configurations file
viper.AddConfigPath(".")
// Enable VIPER to read Environment Variables
viper.AutomaticEnv()
viper.SetConfigType("yml")
var configuration c.Configurations
if err := viper.ReadInConfig(); err != nil {
fmt.Printf("Error reading config file, %s", err)
}
err := viper.Unmarshal(&configuration)
if err != nil {
fmt.Printf("Unable to decode into struct, %v", err)
}
ctx, _ := context.WithTimeout(context.Background(), 10*time.Second)
clientOptions := options.Client().ApplyURI(fmt.Sprintf("mongodb+srv://%s:%s#cluster0.ucnph.mongodb.net/%s?retryWrites=true&w=majority", configuration.Database.DBUser, configuration.Database.DBPass, configuration.Database.DBName))
port := fmt.Sprintf(":%d", configuration.Server.Port)
mongo.Connect(ctx, clientOptions)
router := mux.NewRouter()
router.HandleFunc("/api/admin", admin.CreateAdminEndpoint).Methods("POST")
router.HandleFunc("/api/admin/login", admin.AdminLoginEndpoint).Methods("POST")
router.HandleFunc("/api/tournament/getfromweb", tournament.GetTournamentData).Methods("GET")
fmt.Printf("server listening on http://localhost%v", port)
http.ListenAndServe(port, router)
}
Now, in order to manage my code more concisely, I set up modules (as you can see in the imports on main) to handle the functions that mux will use for the endpoints.
On one specific case (handling the "/api/admin" endpoint:
package admin
import (
"context"
"encoding/base64"
"encoding/json"
"fmt"
"io"
"log"
"net/http"
"time"
"github.com/dgrijalva/jwt-go"
c "github.com/moonlightfight/elo-backend/config"
m "github.com/moonlightfight/elo-backend/models"
"github.com/spf13/viper"
"go.mongodb.org/mongo-driver/bson/primitive"
"go.mongodb.org/mongo-driver/mongo"
"golang.org/x/crypto/bcrypt"
)
var client *mongo.Client
// other code here
func CreateAdminEndpoint(response http.ResponseWriter, request *http.Request) {
response.Header().Set("content-type", "application/json")
var admin m.Admin
err := json.NewDecoder(request.Body).Decode(&admin)
if err != nil {
log.Println(err)
}
// encrypt user password
admin.Password = HashPassword(admin.Password)
fmt.Println(client)
collection := client.Database("test").Collection("Admin")
ctx, ctxErr := context.WithTimeout(context.Background(), 5*time.Second)
if ctxErr != nil {
log.Println(ctxErr)
}
result, resErr := collection.InsertOne(ctx, admin)
if resErr != nil {
log.Println(resErr)
}
json.NewEncoder(response).Encode(result)
}
And when ran, I receive the following error:
2021/06/05 02:02:39 http: panic serving [::1]:53359: runtime error: invalid memory address or nil pointer dereference
This points to the line where I define collection in the endpoint function, which logged as having a nil value. I am clearly not getting the mongo client properly defined in the module, and not sure of the best practice for maintaining this client connection across multiple modules.
The standard way of doing this while avoiding globals would be to define a struct that represents your server, and its methods would be the handlers. Then the methods share the struct's data, and you place things like your mongo client in there.
Something like this (in your admin package):
type Server struct {
client *mongo.Client
}
func NewServer(client *mongo.Client) *Server {
return &Server{client: client}
}
func (srv *Server) CreateAdminEndpoint(response http.ResponseWriter, request *http.Request) {
// ...
// use srv.client here
//
}
And now in main, you create the server like this:
client, err := mongo.Connect(...)
// handle err!
srv := admin.NewServer(client)
router := mux.NewRouter()
router.HandleFunc("/api/admin", srv.CreateAdminEndpoint).Methods("POST")

Dealing with context when reusing a mongodb connection

I'm making multiple goroutines share a single connection by passing client as an argument.
uri := "mongodb://localhost:27017"
ctx := context.Background()
client, err := mongo.Connect(ctx, options.Client().ApplyURI(uri))
go Foo(client)
go Bar(client)
func Foo(client *mongo.Client) {
// ...
}
func Bar(client *mongoClient) {
// ...
}
I'm confused about what to do with ctx. Should I create a new context everytime I query to the database, or should I reuse context just like the client?
It depends on how your Foo and Bar methods behave. Let's imagine the Foo method is a simple short-lived goroutine that makes one query to DB and the only thing you want is to check if its parent context is not Done or Cancelled. Then you can provide parent context to your Foo method.
func main() {
uri := "mongodb://localhost:27017"
ctx := context.Background()
client, err := Connect(ctx, uri)
ctx, cancel := context.WithCancel(ctx)
if err != nil {
panic(err)
}
go Foo(ctx, client)
go Bar(context.WithValue(ctx, "uri", uri), client)
// cancel parent context
cancel()
time.Sleep(5*time.Second)
}
func Foo(ctx context.Context, client *Client) {
fmt.Printf("Foo: %s\n", ctx.Value("uri"))
select {
case <- ctx.Done():
err := ctx.Err()
if err != nil {
// you could switch for the actual reason
fmt.Println("In our case context canceled: ", err)
return
}
fmt.Printf("Do something...")
}
}
On the other hand, if Bar performs some non-trivial logic and makes more than one call to DB you probably want a separate context to be able to cancel it separately from your parent context. Then you could derive a new context from your parent.
func Bar(ctx context.Context, client *Client) {
// Bar has a non trivial logic and needs a separate cancellation and handling
ctx, cancelFunc := context.WithCancel(ctx)
fmt.Printf("Bar: %s\n", ctx.Value("uri"))
// cancel derived context
cancelFunc()
}
I have also done it like this
type DB struct {
client *mongo.Client
}
func (db *DB) GetVideoStream {}
func main() {
ctx, _ := context.WithTimeout(context.Background(), 60*time.Second)
client, err := mongo.Connect(ctx, clientOpts)
db := &DB{client: client}
go db.GetVideoStream()
http.HandleFunc("/api/", db.GetVideoStream)
}
You can use pointer receivers to do the same thing.
I am new still new to the language

Why I get an error "client disconnected" when trying to get documents from mongo collection in go?

I have mongo capped collection and a simple API, written on Go. I built and run it. When I try to sent Get request or simply go localhost:8000/logger in browser - my process closes. Debug shows this happens, while executing "find" in collection. It produces error "client is disconnected". Collection has 1 document, and debug shows it is connected with my helper.
Go version 1.13
My code:
func main() {
r := mux.NewRouter()
r.HandleFunc("/logger", getDocs).Methods("GET")
r.HandleFunc("/logger", createDoc).Methods("POST")
log.Fatal(http.ListenAndServe("localhost:8000", r))
}
func getDocs(w http.ResponseWriter, r *http.Request) {
w.Header().Set("Content-Type", "application/json")
var docs []models.Logger
//Connection mongoDB with helper class
collection := helper.ConnectDB()
cur, err := collection.Find(context.TODO(), bson.M{})
if err != nil {
helper.GetError(err, w)
return
}
defer cur.Close(context.TODO())
for cur.Next(context.TODO()) {
var doc models.Logger
err := cur.Decode(&doc)
if err != nil {
log.Fatal(err)
}
docs = append(docs, doc)
}
if err := cur.Err(); err != nil {
log.Fatal(err)
}
json.NewEncoder(w).Encode(docs)
}
func ConnectDB() *mongo.Collection {
client, err := mongo.NewClient(options.Client().ApplyURI("mongodb://127.0.0.1:27017"))
if err != nil {
log.Fatal(err)
}
fmt.Println("Connected to MongoDB!")
logCollection := client.Database("local").Collection("loggerCollection")
return logCollection
}
According to the documentation, the call to mongo.NewClient doesn't ensure that you can connect the Mongo server. You should first call mongo.Client.Ping() to verify if you can connect to the database or not.
client, err := mongo.NewClient(options.Client().ApplyURI("mongodb://127.0.0.1:27017"))
if err != nil {
log.Fatal(err)
}
if err := client.Ping(context.TODO(), readpref.Primary()); err != nil {
// Can't connect to Mongo server
log.Fatal(err)
}
There could be several reasons behind failing to connect, the most obvious one is incorrect setup of ports. Is your mongodb server up and listening on port 27017? Is there any change you're running mongodb with Docker and it's not forwarding to the correct port?
I faced similar issue , read #Jay answer it definitely helped , as I checked my MongoDB was running using "MongoDB Compass" , then I changed the location of my insert statement , previously I was calling before the call of "context.WithTimeout". Below is working code.
package main
import (
"context"
"log"
"time"
"go.mongodb.org/mongo-driver/mongo"
"go.mongodb.org/mongo-driver/mongo/options"
)
type Book struct {
Name string `json:"name,omitempty"`
PublisherID string `json:"publisherid,omitempty"`
Cost string `json:"cost,omitempty"`
StartTime string `json:"starttime,omitempty"`
EndTime string `json:"endtime,omitempty"`
}
func main() {
client, err := mongo.NewClient(options.Client().ApplyURI("mongodb://localhost:27017"))
if err != nil {
log.Fatal(err)
}
ctx, _ := context.WithTimeout(context.Background(), 10*time.Second)
err = client.Connect(ctx)
if err != nil {
log.Fatal(err)
}
defer client.Disconnect(ctx)
testCollection := client.Database("BooksCollection").Collection("BooksRead")
inserRes, err := testCollection.InsertOne(context.TODO(), Book{Name: "Harry Potter", PublisherID: "IBN123", Cost: "1232", StartTime: "2013-10-01T01:11:18.965Z", EndTime: "2013-10-01T01:11:18.965Z"})
log.Println("InsertResponse : ", inserRes)
log.Println("Error : ", err)
}
I can see document inserted in console as well as in "MongoDB Comapass."
In heiper function "ConnectDB" after "NewClient" I must use "client.Connect(context.TODO())"
before any other use of client

Re-creating mgo sessions in case of errors (read tcp 127.0.0.1:46954->127.0.0.1:27017: i/o timeout)

I wonder about MongoDB session management in Go using mgo, especially about how to correctly ensure a session is closed and how to react on write failures.
I have read the following:
Best practice to maintain a mgo session
Should I copy session for each operation in mgo?
Still, cannot apply it to my situation.
I have two goroutines which store event after event into MongoDB sharing the same *mgo.Session, both looking essiantially like the following:
func storeEvents(session *mgo.Session) {
session_copy := session.Copy()
// *** is it correct to defer the session close here? <-----
defer session_copy.Close()
col := session_copy.DB("DB_NAME").C("COLLECTION_NAME")
for {
event := GetEvent()
err := col.Insert(&event)
if err != nil {
// *** insert FAILED - how to react properly? <-----
session_copy = session.Copy()
defer session_copy.Close()
}
}
}
col.Insert(&event) after some hours returns the error
read tcp 127.0.0.1:46954->127.0.0.1:27017: i/o timeout
and I am unsure how to react on this. After this error occurs, it occurs on all subsequent writes, hence it seems I have to create a new session. Alternatives for me seem:
1) restart the whole goroutine, i.e.
if err != nil {
go storeEvents(session)
return
}
2) create a new session copy
if err != nil {
session_copy = session.Copy()
defer session_copy.Close()
col := session_copy.DB("DB_NAME").C("COLLECTION_NAME")
continue
}
--> Is it correct how I use defer session_copy.Close()? (Note the above defer references the Close() function of another session. Anyway, those sessions will never be closed since the function never returns. I.e., with time, many sessions will be created and not closed.
Other options?
So I don't know if this is going to help you any, but I don't have any issues with this set up.
I have a mongo package that I import from. This is a template of my mongo.go file
package mongo
import (
"time"
"gopkg.in/mgo.v2"
)
var (
// MyDB ...
MyDB DataStore
)
// create the session before main starts
func init() {
MyDB.ConnectToDB()
}
// DataStore containing a pointer to a mgo session
type DataStore struct {
Session *mgo.Session
}
// ConnectToTagserver is a helper method that connections to pubgears' tagserver
// database
func (ds *DataStore) ConnectToDB() {
mongoDBDialInfo := &mgo.DialInfo{
Addrs: []string{"ip"},
Timeout: 60 * time.Second,
Database: "db",
}
sess, err := mgo.DialWithInfo(mongoDBDialInfo)
if err != nil {
panic(err)
}
sess.SetMode(mgo.Monotonic, true)
MyDB.Session = sess
}
// Close is a helper method that ensures the session is properly terminated
func (ds *DataStore) Close() {
ds.Session.Close()
}
Then in another package, for example main Updated Based on the comment below
package main
import (
"../models/mongo"
)
func main() {
// Grab the main session which was instantiated in the mongo package init function
sess := mongo.MyDB.Session
// pass that session in
storeEvents(sess)
}
func storeEvents(session *mgo.Session) {
session_copy := session.Copy()
defer session_copy.Close()
// Handle panics in a deferred fuction
// You can turn this into a wrapper (middleware)
// remove this this function, and just wrap your calls with it, using switch cases
// you can handle all types of errors
defer func(session *mgo.Session) {
if err := recover(); err != nil {
fmt.Printf("Mongo insert has caused a panic: %s\n", err)
fmt.Println("Attempting to insert again")
session_copy := session.Copy()
defer session_copy.Close()
col := session_copy.DB("DB_NAME").C("COLLECTION_NAME")
event := GetEvent()
err := col.Insert(&event)
if err != nil {
fmt.Println("Attempting to insert again failed")
return
}
fmt.Println("Attempting to insert again succesful")
}
}(session)
col := session_copy.DB("DB_NAME").C("COLLECTION_NAME")
event := GetEvent()
err := col.Insert(&event)
if err != nil {
panic(err)
}
}
I use a similar setup on my production servers on AWS. I do over 1 million inserts an hour. Hope this helps. Another things I've done to ensure that the mongo servers can handle the connections is increate the ulimit on my production machines. It's talked about in this stack