Configuring mongo with gin golang framework - mongodb

I'm trying to configure mongo in my Go app. I'm using the Gin framework. I'm also using the mgo V2 driver for mongo. I want to connect to mongo as a middlware. Here's what I got:
func Run(cfg common.Config) error {
doWorkResource := &DoWorkResource{db: dbmap}
r := gin.New()
r.Use(middleware.DB())
r.POST("/register", doWorkResource.Register)
r.POST("/login", doWorkResource.Login)
r.Run(cfg.SvcHost)
return nil
}
This is the DB function:
func DB() gin.HandlerFunc {
session, err := mgo.Dial("localhost:27017")
if err != nil {
panic(err)
}
defer session.Close()
return func(c *gin.Context) {
s := session.Clone()
db := s.DB("testing").C("testData")
err = db.Insert(&Person{"Ale", "+55 53 8116 9639"},
&Person{"Cla", "+55 53 8402 8510"})
if err != nil {
log.Fatal(err)
}
c.Next()
}
}
type Person struct {
Name string
Phone string
}
The insertion is executed if I run it directly from the main function, but not if it's run from the DB() function. In fact, I logged from within the return statement in the DB() function and nothing in there is being executed. I have a feeling I need to call it as an argument to one of my endpoints, but when I do that I get
cannot use middleware.DB (type func() gin.HandlerFunc) as type gin.HandlerFunc in argument to r.RouterGroup.POST

Looks like your problem is right here:
defer session.Close()
What you're really doing when you register the middleware is not call that middleware every time a call comes in, but defining what to do every time a call comes in. First you run a initial set of commands. That would be this part:
session, err := mgo.Dial("localhost:27017")
if err != nil {
panic(err)
}
defer session.Close()
And then you return a set of instructions to run every time one of your endpoints gets hit, that would be this part:
s := session.Clone()
db := s.DB("testing").C("testData")
err = db.Insert(&Person{"Ale", "+55 53 8116 9639"},
&Person{"Cla", "+55 53 8402 8510"})
if err != nil {
log.Fatal(err)
}
c.Next()
And when you return from the middleware initializer, it activates the defer statement. Then when the first call comes in and tries to run the instructions you returned in the form of a HandlerFunc it fails because the session it is trying to use has already been closed.

Related

Why I get an error "client disconnected" when trying to get documents from mongo collection in go?

I have mongo capped collection and a simple API, written on Go. I built and run it. When I try to sent Get request or simply go localhost:8000/logger in browser - my process closes. Debug shows this happens, while executing "find" in collection. It produces error "client is disconnected". Collection has 1 document, and debug shows it is connected with my helper.
Go version 1.13
My code:
func main() {
r := mux.NewRouter()
r.HandleFunc("/logger", getDocs).Methods("GET")
r.HandleFunc("/logger", createDoc).Methods("POST")
log.Fatal(http.ListenAndServe("localhost:8000", r))
}
func getDocs(w http.ResponseWriter, r *http.Request) {
w.Header().Set("Content-Type", "application/json")
var docs []models.Logger
//Connection mongoDB with helper class
collection := helper.ConnectDB()
cur, err := collection.Find(context.TODO(), bson.M{})
if err != nil {
helper.GetError(err, w)
return
}
defer cur.Close(context.TODO())
for cur.Next(context.TODO()) {
var doc models.Logger
err := cur.Decode(&doc)
if err != nil {
log.Fatal(err)
}
docs = append(docs, doc)
}
if err := cur.Err(); err != nil {
log.Fatal(err)
}
json.NewEncoder(w).Encode(docs)
}
func ConnectDB() *mongo.Collection {
client, err := mongo.NewClient(options.Client().ApplyURI("mongodb://127.0.0.1:27017"))
if err != nil {
log.Fatal(err)
}
fmt.Println("Connected to MongoDB!")
logCollection := client.Database("local").Collection("loggerCollection")
return logCollection
}
According to the documentation, the call to mongo.NewClient doesn't ensure that you can connect the Mongo server. You should first call mongo.Client.Ping() to verify if you can connect to the database or not.
client, err := mongo.NewClient(options.Client().ApplyURI("mongodb://127.0.0.1:27017"))
if err != nil {
log.Fatal(err)
}
if err := client.Ping(context.TODO(), readpref.Primary()); err != nil {
// Can't connect to Mongo server
log.Fatal(err)
}
There could be several reasons behind failing to connect, the most obvious one is incorrect setup of ports. Is your mongodb server up and listening on port 27017? Is there any change you're running mongodb with Docker and it's not forwarding to the correct port?
I faced similar issue , read #Jay answer it definitely helped , as I checked my MongoDB was running using "MongoDB Compass" , then I changed the location of my insert statement , previously I was calling before the call of "context.WithTimeout". Below is working code.
package main
import (
"context"
"log"
"time"
"go.mongodb.org/mongo-driver/mongo"
"go.mongodb.org/mongo-driver/mongo/options"
)
type Book struct {
Name string `json:"name,omitempty"`
PublisherID string `json:"publisherid,omitempty"`
Cost string `json:"cost,omitempty"`
StartTime string `json:"starttime,omitempty"`
EndTime string `json:"endtime,omitempty"`
}
func main() {
client, err := mongo.NewClient(options.Client().ApplyURI("mongodb://localhost:27017"))
if err != nil {
log.Fatal(err)
}
ctx, _ := context.WithTimeout(context.Background(), 10*time.Second)
err = client.Connect(ctx)
if err != nil {
log.Fatal(err)
}
defer client.Disconnect(ctx)
testCollection := client.Database("BooksCollection").Collection("BooksRead")
inserRes, err := testCollection.InsertOne(context.TODO(), Book{Name: "Harry Potter", PublisherID: "IBN123", Cost: "1232", StartTime: "2013-10-01T01:11:18.965Z", EndTime: "2013-10-01T01:11:18.965Z"})
log.Println("InsertResponse : ", inserRes)
log.Println("Error : ", err)
}
I can see document inserted in console as well as in "MongoDB Comapass."
In heiper function "ConnectDB" after "NewClient" I must use "client.Connect(context.TODO())"
before any other use of client

golang mongo-db transaction cannot create namespace

I am trying to do db transaction with mongo-db in golang but getting cannot create namespace error
// For a replica set, include the replica set name and a seedlist of the members in the URI string; e.g.
// uri := "mongodb://mongodb0.example.com:27017,mongodb1.example.com:27017/?replicaSet=myRepl"
// For a sharded cluster, connect to the mongos instances; e.g.
// uri := "mongodb://mongos0.example.com:27017,mongos1.example.com:27017/"
uri := "mongodb://mongo-0/block-recorder?replicaSet=rs0"
// var uri string
clientOpts := options.Client().ApplyURI(uri)
client, err := mongo.Connect(ctx, clientOpts)
if err != nil {
panic(err)
}
defer func() { _ = client.Disconnect(ctx) }()
// Prereq: Create collections.
wcMajority := writeconcern.New(writeconcern.WMajority(), writeconcern.WTimeout(1*time.Second))
wcMajorityCollectionOpts := options.Collection().SetWriteConcern(wcMajority)
blockCollection := client.Database("block-recorder").Collection("Block", wcMajorityCollectionOpts)
// Step 1: Define the callback that specifies the sequence of operations to perform inside the transaction.
callback := func(sessCtx mongo.SessionContext) (interface{}, error) {
// Important: You must pass sessCtx as the Context parameter to the operations for them to be executed in the
// transaction.
dbBlock := model.TransferBlockData(block)
if _, err := blockCollection.InsertOne(sessCtx, dbBlock); err != nil {
return nil, err
}
return nil, nil
}
// Step 2: Start a session and run the callback using WithTransaction.
session, err := client.StartSession()
if err != nil {
panic(err)
}
defer session.EndSession(ctx)
result, err := session.WithTransaction(ctx, callback)
if err != nil {
panic(err)
}
fmt.Printf("result: %v\n", result)
This is the sample code I am using, Getting below error
panic: multiple write errors: [{write errors: [{Cannot create namespace block-recorder.Block in multi-document transaction.}]}, {<nil>}]
I uses gorm most of the times, This is my first time using mongo with golang, It says create collection first which I am already doing ? Is there any other method to create collection for mongo in golang ?
How to do a proper transaction with struct data example ?
MongoDB server cannot currently create collections in a transaction.
If your application is, say, inserting data into a collection that doesn't exist, the server transparently creates the collection in most cases. But this does not work currently if a transaction is active.
Create the collection ahead of time so that it exists by the time the transaction is executing.
Instantiating a collection object in your application does not actually create a collection. To create a collection, try the equivalent of createCollection in the go driver.

Goroutine opening a new connection to database after each request (sqlx) and ticker

Let's consider the following goroutine:
func main(){
...
go dbGoRoutine()
...
}
And the func:
func dbGoRoutine() {
db, err := sqlx.Connect("postgres", GetPSQLInfo())
if err != nil {
panic(err)
}
defer db.Close()
ticker := time.NewTicker(10 * time.Second)
for _ = range ticker.C {
_, err := db.Queryx("SELECT * FROM table")
if err != nil {
// handle
}
}
}
Each time the function iterates on the ticker it opens a cloudSQL connection
[service... cloudsql-proxy] 2019/11/08 17:05:05 New connection for "location:exemple-db"
I can't figure out why it opens a new connection each time, since the sqlx.Connect is not in the for loop.
This issue is due to how Query function in sql package, it returns Row which are:
Rows is the result of a query. Its cursor starts before the first row of the result set.
Those cursor are stored using cache.
try using Exec().

How to serve a file from mongodb with golang

I am working on a go project where I need to serve files stored in mongodb. The files are stored in a GridFs. I use gopkg.in/mgo.v2 as package to connect and query the db.
I can retrieve the file from the db, that is not hard.
f, err := s.files.OpenId(id)
But how can I serve that file with http?
I work with the JulienSchmidt router to handle all the other restfull requests.
The solutions I find always use static files, not files from a db.
Thanks in advance
Tip: Recommended to use github.com/globalsign/mgo instead of gopkg.in/mgo.v2 (the latter is not maintained anymore).
The mgo.GridFile type implements io.Reader, so you could use io.Copy() to copy its content into the http.ResponseWriter.
But since mgo.GridFile also implements io.Seeker, you may take advantage of http.ServeContent(). Quoting its doc:
The main benefit of ServeContent over io.Copy is that it handles Range requests properly, sets the MIME type, and handles If-Match, If-Unmodified-Since, If-None-Match, If-Modified-Since, and If-Range requests.
Example handler serving a file:
func serveFromDB(w http.ResponseWriter, r *http.Request) {
var gridfs *mgo.GridFS // Obtain GridFS via Database.GridFS(prefix)
name := "somefile.pdf"
f, err := gridfs.Open(name)
if err != nil {
log.Printf("Failed to open %s: %v", name, err)
http.Error(w, "something went wrong", http.StatusInternalServerError)
return
}
defer f.Close()
http.ServeContent(w, r, name, time.Now(), f) // Use proper last mod time
}
its old but i got another solution with goMongo driver by importing
"go.mongodb.org/mongo-driver/mongo/gridfs"
var bucket *gridfs.Bucket //creates a bucket
dbConnection, err := db.GetDBCollection() //connect db with your your
if err != nil {
log.Fatal(err)
}
bucket, err = gridfs.NewBucket(dbConnection)
if err != nil {
log.Fatal(err)
}
name := "br100_update.txt"
downloadStream, err := bucket.OpenDownloadStreamByName(name)
if err != nil {
log.Printf("Failed to open %s: %v", name, err)
http.Error(w, "something went wrong", http.StatusInternalServerError)
return
}
defer func() {
if err := downloadStream.Close(); err != nil {
log.Fatal(err)
}
}()
// Use SetReadDeadline to force a timeout if the download does not succeed in
// 2 seconds.
if err = downloadStream.SetReadDeadline(time.Now().Add(2 * time.Second)); err
!= nil {
log.Fatal(err)
}
// this code below use to read the file
fileBuffer := bytes.NewBuffer(nil)
if _, err := io.Copy(fileBuffer, downloadStream); err != nil {
log.Fatal(err)
}

Re-creating mgo sessions in case of errors (read tcp 127.0.0.1:46954->127.0.0.1:27017: i/o timeout)

I wonder about MongoDB session management in Go using mgo, especially about how to correctly ensure a session is closed and how to react on write failures.
I have read the following:
Best practice to maintain a mgo session
Should I copy session for each operation in mgo?
Still, cannot apply it to my situation.
I have two goroutines which store event after event into MongoDB sharing the same *mgo.Session, both looking essiantially like the following:
func storeEvents(session *mgo.Session) {
session_copy := session.Copy()
// *** is it correct to defer the session close here? <-----
defer session_copy.Close()
col := session_copy.DB("DB_NAME").C("COLLECTION_NAME")
for {
event := GetEvent()
err := col.Insert(&event)
if err != nil {
// *** insert FAILED - how to react properly? <-----
session_copy = session.Copy()
defer session_copy.Close()
}
}
}
col.Insert(&event) after some hours returns the error
read tcp 127.0.0.1:46954->127.0.0.1:27017: i/o timeout
and I am unsure how to react on this. After this error occurs, it occurs on all subsequent writes, hence it seems I have to create a new session. Alternatives for me seem:
1) restart the whole goroutine, i.e.
if err != nil {
go storeEvents(session)
return
}
2) create a new session copy
if err != nil {
session_copy = session.Copy()
defer session_copy.Close()
col := session_copy.DB("DB_NAME").C("COLLECTION_NAME")
continue
}
--> Is it correct how I use defer session_copy.Close()? (Note the above defer references the Close() function of another session. Anyway, those sessions will never be closed since the function never returns. I.e., with time, many sessions will be created and not closed.
Other options?
So I don't know if this is going to help you any, but I don't have any issues with this set up.
I have a mongo package that I import from. This is a template of my mongo.go file
package mongo
import (
"time"
"gopkg.in/mgo.v2"
)
var (
// MyDB ...
MyDB DataStore
)
// create the session before main starts
func init() {
MyDB.ConnectToDB()
}
// DataStore containing a pointer to a mgo session
type DataStore struct {
Session *mgo.Session
}
// ConnectToTagserver is a helper method that connections to pubgears' tagserver
// database
func (ds *DataStore) ConnectToDB() {
mongoDBDialInfo := &mgo.DialInfo{
Addrs: []string{"ip"},
Timeout: 60 * time.Second,
Database: "db",
}
sess, err := mgo.DialWithInfo(mongoDBDialInfo)
if err != nil {
panic(err)
}
sess.SetMode(mgo.Monotonic, true)
MyDB.Session = sess
}
// Close is a helper method that ensures the session is properly terminated
func (ds *DataStore) Close() {
ds.Session.Close()
}
Then in another package, for example main Updated Based on the comment below
package main
import (
"../models/mongo"
)
func main() {
// Grab the main session which was instantiated in the mongo package init function
sess := mongo.MyDB.Session
// pass that session in
storeEvents(sess)
}
func storeEvents(session *mgo.Session) {
session_copy := session.Copy()
defer session_copy.Close()
// Handle panics in a deferred fuction
// You can turn this into a wrapper (middleware)
// remove this this function, and just wrap your calls with it, using switch cases
// you can handle all types of errors
defer func(session *mgo.Session) {
if err := recover(); err != nil {
fmt.Printf("Mongo insert has caused a panic: %s\n", err)
fmt.Println("Attempting to insert again")
session_copy := session.Copy()
defer session_copy.Close()
col := session_copy.DB("DB_NAME").C("COLLECTION_NAME")
event := GetEvent()
err := col.Insert(&event)
if err != nil {
fmt.Println("Attempting to insert again failed")
return
}
fmt.Println("Attempting to insert again succesful")
}
}(session)
col := session_copy.DB("DB_NAME").C("COLLECTION_NAME")
event := GetEvent()
err := col.Insert(&event)
if err != nil {
panic(err)
}
}
I use a similar setup on my production servers on AWS. I do over 1 million inserts an hour. Hope this helps. Another things I've done to ensure that the mongo servers can handle the connections is increate the ulimit on my production machines. It's talked about in this stack