Dealing with context when reusing a mongodb connection - mongodb

I'm making multiple goroutines share a single connection by passing client as an argument.
uri := "mongodb://localhost:27017"
ctx := context.Background()
client, err := mongo.Connect(ctx, options.Client().ApplyURI(uri))
go Foo(client)
go Bar(client)
func Foo(client *mongo.Client) {
// ...
}
func Bar(client *mongoClient) {
// ...
}
I'm confused about what to do with ctx. Should I create a new context everytime I query to the database, or should I reuse context just like the client?

It depends on how your Foo and Bar methods behave. Let's imagine the Foo method is a simple short-lived goroutine that makes one query to DB and the only thing you want is to check if its parent context is not Done or Cancelled. Then you can provide parent context to your Foo method.
func main() {
uri := "mongodb://localhost:27017"
ctx := context.Background()
client, err := Connect(ctx, uri)
ctx, cancel := context.WithCancel(ctx)
if err != nil {
panic(err)
}
go Foo(ctx, client)
go Bar(context.WithValue(ctx, "uri", uri), client)
// cancel parent context
cancel()
time.Sleep(5*time.Second)
}
func Foo(ctx context.Context, client *Client) {
fmt.Printf("Foo: %s\n", ctx.Value("uri"))
select {
case <- ctx.Done():
err := ctx.Err()
if err != nil {
// you could switch for the actual reason
fmt.Println("In our case context canceled: ", err)
return
}
fmt.Printf("Do something...")
}
}
On the other hand, if Bar performs some non-trivial logic and makes more than one call to DB you probably want a separate context to be able to cancel it separately from your parent context. Then you could derive a new context from your parent.
func Bar(ctx context.Context, client *Client) {
// Bar has a non trivial logic and needs a separate cancellation and handling
ctx, cancelFunc := context.WithCancel(ctx)
fmt.Printf("Bar: %s\n", ctx.Value("uri"))
// cancel derived context
cancelFunc()
}

I have also done it like this
type DB struct {
client *mongo.Client
}
func (db *DB) GetVideoStream {}
func main() {
ctx, _ := context.WithTimeout(context.Background(), 60*time.Second)
client, err := mongo.Connect(ctx, clientOpts)
db := &DB{client: client}
go db.GetVideoStream()
http.HandleFunc("/api/", db.GetVideoStream)
}
You can use pointer receivers to do the same thing.
I am new still new to the language

Related

How should Go Context be declared?

Suppose I have a file that modifies database, should the functions share a single context or should each function have their own context?
Sharing context
var (
ctx = context.Background()
)
func test1() {
res, err := Collection.InsertOne(ctx, data)
}
func test2() {
res, err := Collection.InsertOne(ctx, data)
}
Or should it be like this?
func test1() {
res, err := Collection.InsertOne(context.Background(), data)
}
func test2() {
res, err := Collection.InsertOne(context.Background(), data)
}
Neither. Your context should be request-scoped (for whatever "request" means in your application) and passed down the call chain to each function.
func test1(ctx context.Context) {
res, err := Collection.InsertOne(ctx, data)
}
func test2(ctx context.Context) {
res, err := Collection.InsertOne(ctx, data)
}
If you're building a web server, as indicated in comments, you'll usually get your context from the HTTP request in your handler:
func myHandler(w http.ResponseWriter, r *http.Request) {
ctx := r.Context()
// now pass ctx to any fuction that needs it. This way, if the HTTP
// client/browser cancels the request, your downstream functions will
// be able to abort immediately, without wasting time finishing their
// work.
}
You should not use the first approach. Context is something to be passed down to functions, it should not be declared as a global variable.
The second approach can be used at times, especially if there is no request context. However, if multiple calls are handled within a single server context, you should pass down the context for that call to all the other calls getting a context, so when the context is canceled or expired, all calls would fail.

How to Manage MongoDB Client across Packages in Golang

I am currently starting out with MongoDB on GoLang. My current use case is this.
I wish to initialize connection to my MongoDB Database in a particular package and use the returned client across several other local packages. Here is what I have tried,
I have initialized the connection to MongoDB inside a package called dataLayer as below
package dataLayer
import (
"context"
"log"
"time"
"go.mongodb.org/mongo-driver/mongo"
"go.mongodb.org/mongo-driver/mongo/options"
"go.mongodb.org/mongo-driver/mongo/readpref"
)
func InitDataLayer() {
ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
defer cancel()
client, err := mongo.Connect(ctx, options.Client().ApplyURI(credentials.MONGO_DB_ATLAS_URI))
if err != nil {
log.Fatal(err)
} else {
log.Println("Connected to Database")
}
}
Now if I wish to use the returned client in other packages, is it production safe to keep on calling initDataLayer over and over again to get back a client?
Thanks alot.
You don't have to call InitDataLayer over and over again. You need to create only one client, you can use the same client to connect to Mongo DB from various places.
Mongo client internally maintains a pool of connections, so you don't have to create this client again and again.
It's a good design to store this client as the struct field and keep reusing the same.
Edit:
Create Connection
package dataLayer
import (
"context"
"log"
"time"
"go.mongodb.org/mongo-driver/mongo"
"go.mongodb.org/mongo-driver/mongo/options"
"go.mongodb.org/mongo-driver/mongo/readpref"
)
func InitDataLayer()(*mongo.Client) {
ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
defer cancel()
client, err := mongo.Connect(ctx, options.Client().ApplyURI(credentials.MONGO_DB_ATLAS_URI))
if err != nil {
log.Fatal(err)
} else {
log.Println("Connected to Database")
}
return client
}
main.go,
Close connection at the end of usage, otherwise connection would be leaked.
func main(){
//...
client = dataLayer.InitDataLayer()
defer client.Disconnect(context.Background())
...
}
Repository/DAL, store client in Repository struct
Interface
type BookRepository interface {
FindById( ctx context.Context, id int) (*Book, error)
}
Repository Implementation that store client in struct
type bookRepository struct {
client *mongo.Client
}
func (r *bookRepository) FindById( ctx context.Context, id int) (*Book, error) {
var book Book
err := r.client.DefaultDatabase().Collection("books").FindOne(ctx, bson.M{"_id": id }).Decode(&book)
if err == mongo.ErrNoDocuments {
return nil, nil
}
if err != nil {
return nil, err
}
return &book, nil
}

Golang - Scaling a websocket client for multiple connections to different servers

I have a websocket client. In reality, it is far more complex than the basic code shown below.
I now need to scale this client code to open connections to multiple servers. Ultimately, the tasks that need to be performed when a message is received from the servers is identical.
What would be the best approach to handle this?
As I said above the actual code performed when receiving the message is far more complex than shown in the example.
package main
import (
"flag"
"log"
"net/url"
"os"
"os/signal"
"time"
"github.com/gorilla/websocket"
)
var addr = flag.String("addr", "localhost:1234", "http service address")
func main() {
flag.Parse()
log.SetFlags(0)
interrupt := make(chan os.Signal, 1)
signal.Notify(interrupt, os.Interrupt)
// u := url.URL{Scheme: "ws", Host: *addr, Path: "/echo"}
u := url.URL{Scheme: "ws", Host: *addr, Path: "/"}
log.Printf("connecting to %s", u.String())
c, _, err := websocket.DefaultDialer.Dial(u.String(), nil)
if err != nil {
log.Fatal("dial:", err)
}
defer c.Close()
done := make(chan struct{})
go func() {
defer close(done)
for {
_, message, err := c.ReadMessage()
if err != nil {
log.Println("read:", err)
return
}
log.Printf("recv: %s", message)
}
}()
ticker := time.NewTicker(time.Second)
defer ticker.Stop()
for {
select {
case <-done:
return
case t := <-ticker.C:
err := c.WriteMessage(websocket.TextMessage, []byte(t.String()))
if err != nil {
log.Println("write:", err)
return
}
case <-interrupt:
log.Println("interrupt")
// Cleanly close the connection by sending a close message and then
// waiting (with timeout) for the server to close the connection.
err := c.WriteMessage(websocket.CloseMessage, websocket.FormatCloseMessage(websocket.CloseNormalClosure, ""))
if err != nil {
log.Println("write close:", err)
return
}
select {
case <-done:
case <-time.After(time.Second):
}
return
}
}
}
Modify the interrupt handling to close a channel on interrupt. This allows multiple goroutines to wait on the event by waiting for the channel to close.
shutdown := make(chan struct{})
interrupt := make(chan os.Signal, 1)
signal.Notify(interrupt, os.Interrupt)
go func() {
<-interrupt
log.Println("interrupt")
close(shutdown)
}()
Move the per-connection code to a function. This code is a copy and paste from the question with two changes: the interrupt channel is replaced with the shutdown channel; the function notifies a sync.WaitGroup when the function is done.
func connect(u string, shutdown chan struct{}, wg *sync.WaitGroup) {
defer wg.Done()
log.Printf("connecting to %s", u)
c, _, err := websocket.DefaultDialer.Dial(u, nil)
if err != nil {
log.Fatal("dial:", err)
}
defer c.Close()
done := make(chan struct{})
go func() {
defer close(done)
for {
_, message, err := c.ReadMessage()
if err != nil {
log.Println("read:", err)
return
}
log.Printf("recv: %s", message)
}
}()
ticker := time.NewTicker(time.Second)
defer ticker.Stop()
for {
select {
case <-done:
return
case t := <-ticker.C:
err := c.WriteMessage(websocket.TextMessage, []byte(t.String()))
if err != nil {
log.Println("write:", err)
return
}
case <-shutdown:
// Cleanly close the connection by sending a close message and then
// waiting (with timeout) for the server to close the connection.
err := c.WriteMessage(websocket.CloseMessage, websocket.FormatCloseMessage(websocket.CloseNormalClosure, ""))
if err != nil {
log.Println("write close:", err)
return
}
select {
case <-done:
case <-time.After(time.Second):
}
return
}
}
}
Declare a sync.WaitGroup in main(). For each websocket endpoint that you want to connect to, increment the WaitGroup and start a goroutine to connect that endpoint. After starting the goroutines, wait on the WaitGroup for the goroutines to complete.
var wg sync.WaitGroup
for _, u := range endpoints { // endpoints is []string
// where elements are URLs
// of endpoints to connect to.
wg.Add(1)
go connect(u, shutdown, &wg)
}
wg.Wait()
The code above with an edit to make it run against Gorilla's echo example server is posted on the playground.
is the communication with every different server completely independendant of the other servers? if yes i would go around in a fashion like:
in main create a context with a cancellation function
create a waitgroup in main to track fired up goroutines
for every server, add to the waitgroup, fire up a new goroutine from the main function passing the context and the waitgroup references
main goes in a for/select loop listening to for signals and if one arrives calls the cancelfunc and waits on the waitgroup.
main can also listen on a result chan from the goroutines and maybe print the results itself it the goroutines shouldn't do it directly.
every goroutine has as we said has references for the wg, the context and possibly a chan to return results. now the approach splits on if the goroutine must do one and one thing only, or if it needs to do a sequence of things. for the first approach
if only one thing is to be done we follow an approach like the one descripbed here (observe that to be asyncronous he would in turn fire up a new goroutine to perform the DoSomething() step that would return the result on the channel)
That allows it to be able to accept the cancellation signal at any time. it is up to you to determine how non-blocking you want to be and how prompt you want to be to respond to cancellation signals.Also the benefit of having the a context associated being passed to the goroutines is that you can call the Context enabled versions of most library functions. For example if you want your dials to have a timeout of let's say 1 minute, you would create a new context with timeout from the one passed and then DialContext with that. This allows the dial to stop both from a timeout or the parent (the one you created in main) context's cancelfunc being called.
if more things need to be done ,i usually prefer to do one thing with the goroutine, have it invoke a new one with the next step to be performed (passing all the references down the pipeline) and exit.
this approach scales well with cancellations and being able to stop the pipeline at any step as well as support contexts with dealines easily for steps that can take too long.

too many open files in mgo go server

I'm getting these errors in the logs:
Accept error: accept tcp [::]:80: accept4: too many open files;
for a mongodb server on ubuntu, written in go using mgo. They start appearing after it's been running for about a day.
code:
package main
import (
"encoding/json"
"io"
"net/http"
"gopkg.in/mgo.v2/bson"
)
var (
Database *mgo.Database
)
func hello(w http.ResponseWriter, r *http.Request) {
io.WriteString(w, "hello")
}
func setTile(w http.ResponseWriter, r *http.Request) {
var requestJSON map[string]interface{}
err := json.NewDecoder(r.Body).Decode(&requestJSON)
if err != nil {
http.Error(w, err.Error(), 400)
return
}
collection := Database.C("tiles")
if requestJSON["tileId"] != nil {
query := bson.M{"tileId": requestJSON["tileId"]}
collection.RemoveAll(query)
collection.Insert(requestJSON)
w.WriteHeader(200)
w.Header().Set("Content-Type", "application/json")
js, _ := json.Marshal(map[string]string{"result": "ok"})
w.Write(js)
} else {
w.WriteHeader(200)
w.Header().Set("Content-Type", "application/json")
w.Write(js)
}
}
func getTile(w http.ResponseWriter, r *http.Request) {
var requestJSON map[string]interface{}
err := json.NewDecoder(r.Body).Decode(&requestJSON)
if err != nil {
http.Error(w, err.Error(), 400)
return
}
collection := Database.C("tiles")
var result []map[string]interface{}
if requestJSON["tileId"] != nil {
query := bson.M{"tileId": requestJSON["tileId"]}
collection.Find(query).All(&result)
}
if len(result) > 0 {
w.WriteHeader(200)
w.Header().Set("Content-Type", "application/json")
js, _ := json.Marshal(result[0])
w.Write(js)
} else {
w.WriteHeader(200)
w.Header().Set("Content-Type", "application/json")
js, _ := json.Marshal(map[string]string{"result": "tile id not found"})
w.Write(js)
}
}
func main() {
session, _ := mgo.Dial("localhost")
Database = session.DB("mapdb")
mux := http.NewServeMux()
mux.HandleFunc("/", hello)
mux.HandleFunc("/setTile", setTile)
mux.HandleFunc("/getTile", getTile)
http.ListenAndServe(":80", mux)
}
Is there something in there that needs closing? Or is it structured wrong in some way?
There seems to be lots of places to set the open file limits, so i'm not sure how to find out what the limits actually are. But it seems like increasing the limit isn't the problem anyway, surely something is being opened on every request and not closed.
This is not how you store and use a MongoDB connection in Go.
You have to store an mgo.Session, not an mgo.Database instance. And whenever you need to interact with the MongoDB, you acquire a copy or a clone of the session (e.g. with Session.Copy() or Session.Clone()), and you close it when you don't need it (preferable using a defer statement). This will ensure you don't leak connections.
You also religiously omit checking for errors, please don't do that. Whatever returns an error, do check it and act on it properly (the least you can do is print / log it).
So basically what you need to do is something like this:
var session *mgo.Session
func init() {
var err error
if session, err = mgo.Dial("localhost"); err != nil {
log.Fatal(err)
}
}
func someHandler(w http.ResponseWriter, r *http.Request) {
sess := session.Copy()
defer sess.Close() // Must close!
c := sess.DB("mapdb").C("tiles")
// Do something with the collection, e.g.
var tile bson.M
if err := c.FindId("someTileID").One(&result); err != nil {
// Tile does not exist, send back error, e.g.:
log.Printf("Tile with ID not found: %v, err: %v", "someTileID", err)
http.NotFound(w, r)
return
}
// Do something with tile
}
See related questions:
mgo - query performance seems consistently slow (500-650ms)
Concurrency in gopkg.in/mgo.v2 (Mongo, Go)
You are missing:
defer r.Body.Close()
Make sure it is used before return statement.

Re-creating mgo sessions in case of errors (read tcp 127.0.0.1:46954->127.0.0.1:27017: i/o timeout)

I wonder about MongoDB session management in Go using mgo, especially about how to correctly ensure a session is closed and how to react on write failures.
I have read the following:
Best practice to maintain a mgo session
Should I copy session for each operation in mgo?
Still, cannot apply it to my situation.
I have two goroutines which store event after event into MongoDB sharing the same *mgo.Session, both looking essiantially like the following:
func storeEvents(session *mgo.Session) {
session_copy := session.Copy()
// *** is it correct to defer the session close here? <-----
defer session_copy.Close()
col := session_copy.DB("DB_NAME").C("COLLECTION_NAME")
for {
event := GetEvent()
err := col.Insert(&event)
if err != nil {
// *** insert FAILED - how to react properly? <-----
session_copy = session.Copy()
defer session_copy.Close()
}
}
}
col.Insert(&event) after some hours returns the error
read tcp 127.0.0.1:46954->127.0.0.1:27017: i/o timeout
and I am unsure how to react on this. After this error occurs, it occurs on all subsequent writes, hence it seems I have to create a new session. Alternatives for me seem:
1) restart the whole goroutine, i.e.
if err != nil {
go storeEvents(session)
return
}
2) create a new session copy
if err != nil {
session_copy = session.Copy()
defer session_copy.Close()
col := session_copy.DB("DB_NAME").C("COLLECTION_NAME")
continue
}
--> Is it correct how I use defer session_copy.Close()? (Note the above defer references the Close() function of another session. Anyway, those sessions will never be closed since the function never returns. I.e., with time, many sessions will be created and not closed.
Other options?
So I don't know if this is going to help you any, but I don't have any issues with this set up.
I have a mongo package that I import from. This is a template of my mongo.go file
package mongo
import (
"time"
"gopkg.in/mgo.v2"
)
var (
// MyDB ...
MyDB DataStore
)
// create the session before main starts
func init() {
MyDB.ConnectToDB()
}
// DataStore containing a pointer to a mgo session
type DataStore struct {
Session *mgo.Session
}
// ConnectToTagserver is a helper method that connections to pubgears' tagserver
// database
func (ds *DataStore) ConnectToDB() {
mongoDBDialInfo := &mgo.DialInfo{
Addrs: []string{"ip"},
Timeout: 60 * time.Second,
Database: "db",
}
sess, err := mgo.DialWithInfo(mongoDBDialInfo)
if err != nil {
panic(err)
}
sess.SetMode(mgo.Monotonic, true)
MyDB.Session = sess
}
// Close is a helper method that ensures the session is properly terminated
func (ds *DataStore) Close() {
ds.Session.Close()
}
Then in another package, for example main Updated Based on the comment below
package main
import (
"../models/mongo"
)
func main() {
// Grab the main session which was instantiated in the mongo package init function
sess := mongo.MyDB.Session
// pass that session in
storeEvents(sess)
}
func storeEvents(session *mgo.Session) {
session_copy := session.Copy()
defer session_copy.Close()
// Handle panics in a deferred fuction
// You can turn this into a wrapper (middleware)
// remove this this function, and just wrap your calls with it, using switch cases
// you can handle all types of errors
defer func(session *mgo.Session) {
if err := recover(); err != nil {
fmt.Printf("Mongo insert has caused a panic: %s\n", err)
fmt.Println("Attempting to insert again")
session_copy := session.Copy()
defer session_copy.Close()
col := session_copy.DB("DB_NAME").C("COLLECTION_NAME")
event := GetEvent()
err := col.Insert(&event)
if err != nil {
fmt.Println("Attempting to insert again failed")
return
}
fmt.Println("Attempting to insert again succesful")
}
}(session)
col := session_copy.DB("DB_NAME").C("COLLECTION_NAME")
event := GetEvent()
err := col.Insert(&event)
if err != nil {
panic(err)
}
}
I use a similar setup on my production servers on AWS. I do over 1 million inserts an hour. Hope this helps. Another things I've done to ensure that the mongo servers can handle the connections is increate the ulimit on my production machines. It's talked about in this stack