I'm using the MongoDB Go Driver in my Go (1.11) server which runs on Google Cloud's App Engine. I'm not really sure if I still manually have to set up connection pooling or if it's already being taken care of out of the box. For example I'm not entirely sure what the context (with timeout) exactly means.
My code looks like this:
package tools
import (
"context"
"time"
"valuation-app/settings"
"go.mongodb.org/mongo-driver/mongo"
"go.mongodb.org/mongo-driver/mongo/options"
)
// ConnectToDB starts a new database connection and returns a reference to it
func ConnectToDB() (*mongo.Database, error) {
settings := settings.Get().Database
ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
defer cancel()
options := options.Client().ApplyURI(settings.URI)
options.SetMaxPoolSize(10)
client, err := mongo.Connect(ctx, options)
if err != nil {
return nil, err
}
return client.Database(settings.DatabaseName), nil
}
From comments by #Alongkorn:
Mongo-go-driver takes care of connection pooling by default(100 connection by default)
You can read the source code here
Related
I am very new to Go world. I have some db functions that I need to test.
So first I have a database.go file that connects to a postgres db:
import (
"fmt"
"gorm.io/driver/postgres"
"gorm.io/gorm"
"os"
)
var DB *gorm.DB
var err error
func Open() error {
dsn := fmt.Sprintf("host=%s user=%s password=%s dbname=%s port=%s sslmode=disable", os.Getenv("HOST"), os.Getenv("USER"),
os.Getenv("PASSWORD"), os.Getenv("DB"), os.Getenv("PORT"))
DB, err = gorm.Open(postgres.Open(dsn), &gorm.Config{})
if err != nil {
return err
}
return nil
}
Then I have a customers.go file with functions that interact with that db:
import (
"customers/cmd/database"
"time"
)
type Customers struct {
ID int
CustomerName string
Active bool
Balance float32
ActiveSince time.Time
}
func Get(id int) (Customers, error) {
var customer Customers
result := database.DB.First(&customer, id)
if result.Error != nil {
return Customers{}, result.Error
} else {
return customer, nil
}
}
This is all running in docker, there is customers container and postgres container. Now the question is how do I test my Get(id int) function? I was researching dockertest but that spins up a different db and my Get function uses the one I specified in database.go. So is there a standard Go way to test these functions?
it is a docker net problem but golang:
you can create a docker net, run both container in one net.doc
or use network --network=host
export postgres container's port to localhost, and customers container link to localhost,-pxx:xx
there is standard way of unit tesing in go. pl refer testing and testify/assert. Unit tests are typically written in xxx_test.go file next to the code.
coming to unit testing of db access layer code, one option would be to have a testenv helper and use it on these lines.
customers_test.go:
package dbaccess
import "testing"
func TestDbAccessLayer(t *testing.T) {
testEnv := testUtils.NewTestEnv()
// testEnv should do the required initialization e.g.
// start any mocked services, connection to database, global logger etc.
if err := testEnv.Start(); err != nil {
t.Fatal(err)
}
// at the end of the test, stop need to reset state
// e.g. clear any entries created by test in db
defer testEnv.Stop()
// add test code
// add required assertion to validate
}
have a separate docker-compose.yml file and use it with docker compose command to start/stop services like postgresdb.
go test command may be used to run the tests. refer to the command docs for details.
Or alternatively, what additional imports do I need?
I'd like to start using Postgres as my main DBMS for some development I am doing, and my research suggests that pgx is the database driver of choice at this time. I have setup a connection package:
package database
import (
"fmt"
_ "github.com/jackc/pgx/v5"
"net/url"
"github.com/jmoiron/sqlx"
log "github.com/sirupsen/logrus"
)
func PgConnect(username, password, host, app string) *sqlx.DB {
s := fmt.Sprintf("postgres://%s:%s#%s?app+name=%s&sslmode=allow", url.QueryEscape(username), url.QueryEscape(password), host, app)
db, err := sqlx.Connect("postgres", s)
if err != nil {
log.Panicf("failed to connect to sqlserver://%s:%s#%s err: %s", username, "xxxxxxxxxxxx", host, err)
}
return db
}
The URL evaluates to:
postgres://user:password#database-host:5432/database-name?app+name=app-name&sslmode=allow
and I've also tried the prefix of postgresql here because I've seen both in places on the internet.
For the driver name in the sqlx.Connect("postgres", s) call I have tried postgres, postgresql and pgx.
In all cases the call to connect fails with the error:
sql: unknown driver "postgres" (forgotten import?)
The same code (with driver mssql and mssql URL) works connection to Microsoft SQL Server.
Can anyone help me get this going? I've found very little on the internet for this combination of language/driver/sqlx/postgres.
From the documentation, we can see that the driver's name is pgx:
A database/sql connection can be established through sql.Open.
db, err := sql.Open("pgx", "postgres://pgx_md5:secret#localhost:5432/pgx_test?sslmode=disable")
if err != nil {
return err
}
So you'll need to do two things:
Use the proper driver name:
db, err := sqlx.Connect("pgx", s)
Import the stdlib compatibility pacakge:
import (
_ "github.com/jackc/pgx/v5/stdlib" // Standard library bindings for pgx
)
First time user of Cadence:
Scenario
I have a cadence server running in my sandbox environment.
Intent is to fetch the workflow status
I am trying to use this cadence client
go.uber.org/cadence/client
on my local host to talk to my sandbox cadence server.
This is my simple code snippet:
var cadClient client.Client
func main() {
wfID := "01ERMTDZHBYCH4GECHB3J692PC" << I got this from cadence-ui
ctx := context.Background()
wf := cadClientlient.GetWorkflow(ctx, wfID,"") <<< Panic hits here
log.Println("Workflow RunID: ",wf.GetID())
}
I am sure getting it wrong because the client does not know how to reach the cadence server.
I referred this https://cadenceworkflow.io/docs/go-client/ to find the correct usage but could not find any reference (possible that I might have missed it).
Any help in how to resolve/implement this, will be of much help
I am not sure what panic you got. Based on the code snippet, it's likely that you haven't initialized the client.
To initialize it, follow the sample code here: https://github.com/uber-common/cadence-samples/blob/master/cmd/samples/common/sample_helper.go#L82
and
https://github.com/uber-common/cadence-samples/blob/aac75c7ca03ec0c184d0f668c8cd0ea13d3a7aa4/cmd/samples/common/factory.go#L113
ch, err := tchannel.NewChannelTransport(
tchannel.ServiceName(_cadenceClientName))
if err != nil {
b.Logger.Fatal("Failed to create transport channel", zap.Error(err))
}
b.Logger.Debug("Creating RPC dispatcher outbound",
zap.String("ServiceName", _cadenceFrontendService),
zap.String("HostPort", b.hostPort))
b.dispatcher = yarpc.NewDispatcher(yarpc.Config{
Name: _cadenceClientName,
Outbounds: yarpc.Outbounds{
_cadenceFrontendService: {Unary: ch.NewSingleOutbound(b.hostPort)},
},
})
if b.dispatcher != nil {
if err := b.dispatcher.Start(); err != nil {
b.Logger.Fatal("Failed to create outbound transport channel: %v", zap.Error(err))
client := workflowserviceclient.New(b.dispatcher.ClientConfig(_cadenceFrontendService))
I have recently upgraded to the newer and offical golang mongo driver for an app I am working on.
All is work prefectly for my local development however when I hook it up and point to my backend server I am getting a 'context deadline exceeded' when calling the client.Ping(...) method.
The old driver code still works fine and I also print out the connection string and can copy and paste this into the compass app and it works without issues.
However for the life of me I cant work out why this new code is return a context timeout. Only different thing is that mongo is running on a non-standard port of 32680 and I am also using the mgm package. However it just using the offical mongo driver under the hood.
Mongo version is: 4.0.12 (locally and remote)
Connection code is here:
// NewClient creates a mongo DateBase connection
func NewClient(cfg config.Mongo) (*Client, error) {
// create database connection string
conStr := fmt.Sprintf("mongodb://%s:%s#%s:%s", cfg.Username, cfg.Password, cfg.Host, cfg.Port)
// set mgm conf ie ctxTimeout value
conf := mgm.Config{CtxTimeout: cfg.CtxTimeout}
// setup mgm / DateBase connection
err := mgm.SetDefaultConfig(&conf, cfg.Database, options.Client().ApplyURI(conStr))
if err != nil {
return nil, errors.Wrapf(err, "failed to connect to mongodb. cfg: %+v. conStr: %+v.", cfg, conStr)
}
// get access to underlying mongodb client driver, db and mgmConfig. Need for adding additional tools like seeding/migrations/etc
mgmCfg, client, db, err := mgm.DefaultConfigs()
if err != nil {
return nil, errors.Wrap(err, "failed to return mgm.DefaultConfigs")
}
// NOTE: fails here!
if err := client.Ping(mgm.Ctx(), readpref.Primary()); err != nil {
return nil, errors.Wrapf(err, "Ping failed to mongodb. cfg: %+v. conStr: %+v. mgmCfg: %+v", cfg, conStr, mgmCfg)
}
return &Client{
cfg: cfg,
mgmCfg: mgmCfg,
client: client,
db: db,
}, nil
}
HELP! I have no idea how I can debug this anymore that I have?
Try adding your authsource in your DSN,
something like
mongodb://USER:PASSWORD#HOST:PORT/DBNAME?authsource=AUTHSOURCE
i have try driver mongodb for a simple connection like on tutorial from this repos github mongo-go-driver i just write client like this :
import (
"fmt"
"github.com/mongodb/mongo-go-driver/mongo"
"github.com/mongodb/mongo-go-driver/mongo/options"
)
var client *mongo.Client
func main() {
fmt.Println("Starting the application...")
client, err := mongo.NewClient(options.Client().ApplyURI("mongodb://localhost:27017"))
if err != nil {
fmt.Println(err.Error())
return
}
}
but show error :
cannot use "github.com/mongodb/mongo-go-driver/mongo/options".Client().ApplyURI("mongodb://localhost:27017") (type *"github.com/mongodb/mongo-go-driver/mongo/options".ClientOptions) as type *"go.mongodb.org/mongo-driver/mongo/options".ClientOptions in argument to mongo.NewClient
Driver version use V1.0.0
Any sugestions ?
you should not use github version of mongo driver. It is just fork of repo located here go.mongodb.org/mongo-driver/mongo. So in first place go get go.mongodb.org/mongo-driver/mongo and then your improts you should change
github.com/mongodb/mongo-go-driver/*
to
go.mongodb.org/mongo-driver/*