I have recently upgraded to the newer and offical golang mongo driver for an app I am working on.
All is work prefectly for my local development however when I hook it up and point to my backend server I am getting a 'context deadline exceeded' when calling the client.Ping(...) method.
The old driver code still works fine and I also print out the connection string and can copy and paste this into the compass app and it works without issues.
However for the life of me I cant work out why this new code is return a context timeout. Only different thing is that mongo is running on a non-standard port of 32680 and I am also using the mgm package. However it just using the offical mongo driver under the hood.
Mongo version is: 4.0.12 (locally and remote)
Connection code is here:
// NewClient creates a mongo DateBase connection
func NewClient(cfg config.Mongo) (*Client, error) {
// create database connection string
conStr := fmt.Sprintf("mongodb://%s:%s#%s:%s", cfg.Username, cfg.Password, cfg.Host, cfg.Port)
// set mgm conf ie ctxTimeout value
conf := mgm.Config{CtxTimeout: cfg.CtxTimeout}
// setup mgm / DateBase connection
err := mgm.SetDefaultConfig(&conf, cfg.Database, options.Client().ApplyURI(conStr))
if err != nil {
return nil, errors.Wrapf(err, "failed to connect to mongodb. cfg: %+v. conStr: %+v.", cfg, conStr)
}
// get access to underlying mongodb client driver, db and mgmConfig. Need for adding additional tools like seeding/migrations/etc
mgmCfg, client, db, err := mgm.DefaultConfigs()
if err != nil {
return nil, errors.Wrap(err, "failed to return mgm.DefaultConfigs")
}
// NOTE: fails here!
if err := client.Ping(mgm.Ctx(), readpref.Primary()); err != nil {
return nil, errors.Wrapf(err, "Ping failed to mongodb. cfg: %+v. conStr: %+v. mgmCfg: %+v", cfg, conStr, mgmCfg)
}
return &Client{
cfg: cfg,
mgmCfg: mgmCfg,
client: client,
db: db,
}, nil
}
HELP! I have no idea how I can debug this anymore that I have?
Try adding your authsource in your DSN,
something like
mongodb://USER:PASSWORD#HOST:PORT/DBNAME?authsource=AUTHSOURCE
Related
I am very new to Go world. I have some db functions that I need to test.
So first I have a database.go file that connects to a postgres db:
import (
"fmt"
"gorm.io/driver/postgres"
"gorm.io/gorm"
"os"
)
var DB *gorm.DB
var err error
func Open() error {
dsn := fmt.Sprintf("host=%s user=%s password=%s dbname=%s port=%s sslmode=disable", os.Getenv("HOST"), os.Getenv("USER"),
os.Getenv("PASSWORD"), os.Getenv("DB"), os.Getenv("PORT"))
DB, err = gorm.Open(postgres.Open(dsn), &gorm.Config{})
if err != nil {
return err
}
return nil
}
Then I have a customers.go file with functions that interact with that db:
import (
"customers/cmd/database"
"time"
)
type Customers struct {
ID int
CustomerName string
Active bool
Balance float32
ActiveSince time.Time
}
func Get(id int) (Customers, error) {
var customer Customers
result := database.DB.First(&customer, id)
if result.Error != nil {
return Customers{}, result.Error
} else {
return customer, nil
}
}
This is all running in docker, there is customers container and postgres container. Now the question is how do I test my Get(id int) function? I was researching dockertest but that spins up a different db and my Get function uses the one I specified in database.go. So is there a standard Go way to test these functions?
it is a docker net problem but golang:
you can create a docker net, run both container in one net.doc
or use network --network=host
export postgres container's port to localhost, and customers container link to localhost,-pxx:xx
there is standard way of unit tesing in go. pl refer testing and testify/assert. Unit tests are typically written in xxx_test.go file next to the code.
coming to unit testing of db access layer code, one option would be to have a testenv helper and use it on these lines.
customers_test.go:
package dbaccess
import "testing"
func TestDbAccessLayer(t *testing.T) {
testEnv := testUtils.NewTestEnv()
// testEnv should do the required initialization e.g.
// start any mocked services, connection to database, global logger etc.
if err := testEnv.Start(); err != nil {
t.Fatal(err)
}
// at the end of the test, stop need to reset state
// e.g. clear any entries created by test in db
defer testEnv.Stop()
// add test code
// add required assertion to validate
}
have a separate docker-compose.yml file and use it with docker compose command to start/stop services like postgresdb.
go test command may be used to run the tests. refer to the command docs for details.
Or alternatively, what additional imports do I need?
I'd like to start using Postgres as my main DBMS for some development I am doing, and my research suggests that pgx is the database driver of choice at this time. I have setup a connection package:
package database
import (
"fmt"
_ "github.com/jackc/pgx/v5"
"net/url"
"github.com/jmoiron/sqlx"
log "github.com/sirupsen/logrus"
)
func PgConnect(username, password, host, app string) *sqlx.DB {
s := fmt.Sprintf("postgres://%s:%s#%s?app+name=%s&sslmode=allow", url.QueryEscape(username), url.QueryEscape(password), host, app)
db, err := sqlx.Connect("postgres", s)
if err != nil {
log.Panicf("failed to connect to sqlserver://%s:%s#%s err: %s", username, "xxxxxxxxxxxx", host, err)
}
return db
}
The URL evaluates to:
postgres://user:password#database-host:5432/database-name?app+name=app-name&sslmode=allow
and I've also tried the prefix of postgresql here because I've seen both in places on the internet.
For the driver name in the sqlx.Connect("postgres", s) call I have tried postgres, postgresql and pgx.
In all cases the call to connect fails with the error:
sql: unknown driver "postgres" (forgotten import?)
The same code (with driver mssql and mssql URL) works connection to Microsoft SQL Server.
Can anyone help me get this going? I've found very little on the internet for this combination of language/driver/sqlx/postgres.
From the documentation, we can see that the driver's name is pgx:
A database/sql connection can be established through sql.Open.
db, err := sql.Open("pgx", "postgres://pgx_md5:secret#localhost:5432/pgx_test?sslmode=disable")
if err != nil {
return err
}
So you'll need to do two things:
Use the proper driver name:
db, err := sqlx.Connect("pgx", s)
Import the stdlib compatibility pacakge:
import (
_ "github.com/jackc/pgx/v5/stdlib" // Standard library bindings for pgx
)
First time user of Cadence:
Scenario
I have a cadence server running in my sandbox environment.
Intent is to fetch the workflow status
I am trying to use this cadence client
go.uber.org/cadence/client
on my local host to talk to my sandbox cadence server.
This is my simple code snippet:
var cadClient client.Client
func main() {
wfID := "01ERMTDZHBYCH4GECHB3J692PC" << I got this from cadence-ui
ctx := context.Background()
wf := cadClientlient.GetWorkflow(ctx, wfID,"") <<< Panic hits here
log.Println("Workflow RunID: ",wf.GetID())
}
I am sure getting it wrong because the client does not know how to reach the cadence server.
I referred this https://cadenceworkflow.io/docs/go-client/ to find the correct usage but could not find any reference (possible that I might have missed it).
Any help in how to resolve/implement this, will be of much help
I am not sure what panic you got. Based on the code snippet, it's likely that you haven't initialized the client.
To initialize it, follow the sample code here: https://github.com/uber-common/cadence-samples/blob/master/cmd/samples/common/sample_helper.go#L82
and
https://github.com/uber-common/cadence-samples/blob/aac75c7ca03ec0c184d0f668c8cd0ea13d3a7aa4/cmd/samples/common/factory.go#L113
ch, err := tchannel.NewChannelTransport(
tchannel.ServiceName(_cadenceClientName))
if err != nil {
b.Logger.Fatal("Failed to create transport channel", zap.Error(err))
}
b.Logger.Debug("Creating RPC dispatcher outbound",
zap.String("ServiceName", _cadenceFrontendService),
zap.String("HostPort", b.hostPort))
b.dispatcher = yarpc.NewDispatcher(yarpc.Config{
Name: _cadenceClientName,
Outbounds: yarpc.Outbounds{
_cadenceFrontendService: {Unary: ch.NewSingleOutbound(b.hostPort)},
},
})
if b.dispatcher != nil {
if err := b.dispatcher.Start(); err != nil {
b.Logger.Fatal("Failed to create outbound transport channel: %v", zap.Error(err))
client := workflowserviceclient.New(b.dispatcher.ClientConfig(_cadenceFrontendService))
I'm using the MongoDB Go Driver in my Go (1.11) server which runs on Google Cloud's App Engine. I'm not really sure if I still manually have to set up connection pooling or if it's already being taken care of out of the box. For example I'm not entirely sure what the context (with timeout) exactly means.
My code looks like this:
package tools
import (
"context"
"time"
"valuation-app/settings"
"go.mongodb.org/mongo-driver/mongo"
"go.mongodb.org/mongo-driver/mongo/options"
)
// ConnectToDB starts a new database connection and returns a reference to it
func ConnectToDB() (*mongo.Database, error) {
settings := settings.Get().Database
ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
defer cancel()
options := options.Client().ApplyURI(settings.URI)
options.SetMaxPoolSize(10)
client, err := mongo.Connect(ctx, options)
if err != nil {
return nil, err
}
return client.Database(settings.DatabaseName), nil
}
From comments by #Alongkorn:
Mongo-go-driver takes care of connection pooling by default(100 connection by default)
You can read the source code here
Hi I am trying to monitor postgres SQL with Prometheus. For this purpose I am using this exporter https://github.com/wrouesnel/postgres_exporter
I am starting the exporter in my docker-compose.yml like this:
exporter-postgres:
image: wrouesnel/postgres_exporter
ports:
- 9113:9113
environment:
- DATA_SOURCE_NAME="postgresql://user:pass#localhost:5432/?sslmode=disable"
When the exporter is trying to access the database errors like this are thrown:
Error running query on database: pg_stat_database pg: Could not detect default username. Please provide one explicitly. file="postgres-exporter.go" line=490
and
Error scanning runtime variable: pg_stat_database pg: Could not detect default username. Please provide one explicitly. file="postgres-exporter.go" line=464
I am not really sure what this message could mean. Also I am not really sure if the issues originates in my docker-compose file, or the exporter.
The lines which throw the error in the postgres-exporter.go are:
// Use SHOW to get the value
row := db.QueryRow(fmt.Sprintf("SHOW %s;", columnName))
var val interface{}
err := row.Scan(&val)
if err != nil {
log.Errorln("Error scanning runtime variable:", columnName, err)
continue
}
and
query, er := queryOverrides[namespace]
if er == false {
query = fmt.Sprintf("SELECT * FROM %s;", namespace)
}
// Don't fail on a bad scrape of one metric
rows, err := db.Query(query)
if err != nil {
log.Println("Error running query on database: ", namespace, err)
e.error.Set(1)
return
}
https://github.com/wrouesnel/postgres_exporter/blob/master/postgres_exporter.go
I am thankful for any help!
Edit:
Here is the connection to the database:
db, err := sql.Open("postgres", e.dsn)
Whereas e.dsn is generated like this:
dsn := os.Getenv("DATA_SOURCE_NAME")
The connection doesn't throw any error
Hey for anyone having a similiar issue in the future:
The problem was this line in the docker-compose.yml
- DATA_SOURCE_NAME="postgresql://user:pass#localhost:5432/?sslmode=disable"
Changing it to
- DATA_SOURCE_NAME=postgresql://user:pass#localhost:5432/?sslmode=disable
(Without the quotes) made everything work :)