I have my database connection in my admin package setup like this,
Template File:
type Template struct{}
func NewAdmin() *Template {
return &Template{}
}
Database File:
type Database struct {
T *Template
}
func (admin *Database) DB() *gorm.DB {
db, err := gorm.Open("postgres", "host=localhost port=5010 user=postgres dbname=postgres password=password sslmode=disable")
if err != nil {
panic(err)
}
return db
}
Now I using that connection in my controllers package, like so
Controller Teamplate
type Template struct {
Connection *admin.Database
}
Profile File:
type ProfilesController struct {
T *Template
}
func (c *ProfilesController) ProfileList(ec echo.Context) error {
profile := []models.Profile{}
c.T.Connection.DB().Find(&profile)
if len(profile) <= 0 {
reply := map[string]string{"Message": "No Profiles Found", "Code": "204"}
return ec.JSON(http.StatusBadRequest, reply)
}
return ec.JSON(http.StatusOK, profile)
}
Now this was all working fine but I have now move on to building the frontend to this api. I am getting pq: sorry, too many clients already after about 96 or so requests.
So I run it though postman and got the same result. This is what I have done to correct the issue,
db := *c.T.Connection.DB()
db.Find(&profile)
defer db.Close()
Now that seems to work, I push over 500 requests though with postman and it worked fine. I am guest its the db.Close() that is helping there.
But I have read that the connection is a pool, so should the orginal code not work without needed a close on the connection? I thought that idle connections were released by the system over it was done with them? I have also read that due to it being a pool its not good to use db.Close().
So I am a little confused? Is what I done to fix the connection issue good? or is there a better way?
Many thanks.
You need to just create the one connection, and return the same instance:
type Database struct {
T *Template
}
var db *gorm.DB
func init() {
var err error
db, err = gorm.Open("postgres", "host=localhost port=5010 user=postgres dbname=postgres password=password sslmode=disable")
if err != nil {
panic(err)
}
}
func (admin *Database) DB() *gorm.DB {
return db
}
Related
I have a set of functions in my web API app. They perform some operations on the data in the Postgres database.
func CreateUser () {
db, err := sql.Open("postgres", "user=postgres password=password dbname=api_dev sslmode=disable")
// Do some db operations here
}
I suppose functions should work with db independently from each other, so now I have sql.Open(...) inside each function. I don't know if it's a correct way to manage db connection.
Should I open it somewhere once the app starts and pass db as an argument to the corresponding functions instead of opening the connection in every function?
Opening a db connection every time it's needed is a waste of resources and it's slow.
Instead, you should create an sql.DB once, when your application starts (or on first demand), and either pass it where it is needed (e.g. as a function parameter or via some context), or simply make it a global variable and so everyone can access it. It's safe to call from multiple goroutines.
Quoting from the doc of sql.Open():
The returned DB is safe for concurrent use by multiple goroutines and maintains its own pool of idle connections. Thus, the Open function should be called just once. It is rarely necessary to close a DB.
You may use a package init() function to initialize it:
var db *sql.DB
func init() {
var err error
db, err = sql.Open("yourdriver", "yourDs")
if err != nil {
log.Fatal("Invalid DB config:", err)
}
}
One thing to note here is that sql.Open() may not create an actual connection to your DB, it may just validate its arguments. To test if you can actually connect to the db, use DB.Ping(), e.g.:
func init() {
var err error
db, err = sql.Open("yourdriver", "yourDs")
if err != nil {
log.Fatal("Invalid DB config:", err)
}
if err = db.Ping(); err != nil {
log.Fatal("DB unreachable:", err)
}
}
I will use a postgres example
package main
import necessary packages and don't forget the postgres driver
import (
"database/sql"
_ "github.com/lib/pq" //postgres driver
)
initialize your connection in the package scope
var db *sql.DB
have an init function for your connection
func init() {
var err error
db, err = sql.open("postgres", "connectionString")
//connectioString example => 'postgres://username:password#localhost/dbName?sslmode=disable'
if err != nil {
panic(err)
}
err = db.Ping()
if err != nil {
panic(err)
}
// note, we haven't deffered db.Close() at the init function since the connection will close after init. you could close it at main or ommit it
}
main function
func main() {
defer db.Close() //optional
//run your db functions
}
checkout this example
https://play.golang.org/p/FAiGbqeJG0H
How do I create a proper mongo based application in Go using the official driver(go.mongodb.org/mongo-driver/mongo)? I have a MongoConnect() and MongoDisconnect(client) function to create a connection and delete it. But, it's not too efficient and starts leaking FD as the app has got around 40 functions and finding all the missed MongoDisconnect() becomes hectic.
The current MongoConnect and MongoDisconnect are as follows.
func MongoConnect() (*mongo.Client, error) {
clientOptions := options.Client().ApplyURI("mongodb://localhost:27017")
client, err := mongo.Connect(context.TODO(), clientOptions)
if err != nil {
Logger(err)
return nil, err
}
err = client.Ping(context.TODO(), nil)
if err != nil {
Logger(err)
return nil, err
}
return client, err
}
func MongoDisconnect(client *mongo.Client) {
_ = client.Disconnect(context.TODO())
}
I am looking for a method that would still use MongoConnect() to create connections and would automatically kill the client without the usage of MongoDisconnect().
PS. Other methods that are better than the above requirement are also welcome
I'm not sure that there is an 'efficient' way to fix the underlying issue, you probably will need to look at all the places where you've called MongoConnect() and ensure you have a corresponding MongoDisconnect().
In saying that, what you might want to look at is implemententing the right pattern for connecting to databases.
Generally speaking, if your database driver takes care of managing connections for you then you should create the connection once and just pass it around as needed.
You could also defer the closing of that connection to a go routine which would close it once it was no longer needed (when you're application is shutting down).
Here is a code snippet of how this is implemented:
// =========================================================================
// Start Database
log.Println("main: Initializing database support")
db, err := database.Open(database.Config{
User: cfg.DB.User,
Password: cfg.DB.Password,
Host: cfg.DB.Host,
Name: cfg.DB.Name,
DisableTLS: cfg.DB.DisableTLS,
})
if err != nil {
return errors.Wrap(err, "connecting to db")
}
defer func() {
log.Printf("main: Database Stopping : %s", cfg.DB.Host)
db.Close()
}()
I didn't write this code and its part of a larger project scaffold which has some other nice patterns for web applications.
Ultimate service
I am expected to have 80% test coverage even for pushing the basic project structure. I am a bit confused how do I write unit tests for the following code to Connect to postgres db and ping postgres for health check. Can someone help me please.
var postgres *sql.DB
// ConnectToPostgres func to connect to postgres
func ConnectToPostgres(connStr string) (*sql.DB, error) {
db, err := sql.Open("postgres", connStr)
if err != nil {
log.Println("postgres-client ", err)
return nil, err
}
postgres = db
return db, nil
}
// PostgresHealthCheck to ping database and check for errors
func PostgresHealthCheck() error {
if err := postgres.Ping(); err != nil {
return err
}
return nil
}
type PostgresRepo struct {
db *sql.DB
}
// NewPostgresRepo constructor
func NewPostgresRepo(database *sql.DB) *PostgresRepo {
return &PostgresRepo{
db: database,
}
}
You need to use this : https://github.com/DATA-DOG/go-sqlmock
Its very easy to use. Here is an example where a controller is getting tested using a mocked SQL :
Implementation
func (up UserProvider) GetUsers() ([]models.User, error) {
var users = make([]models.User, 0, 10)
rows, err := up.DatabaseProvider.Query("SELECT firstname, lastname, email, age FROM Users;")
if err != nil {
return nil, err
}
for rows.Next() {
var u models.User = models.User{}
err := rows.Scan(&u.Name, &u.Lastname, &u.Email, &u.Age)
if err != nil {
return nil, err
}
users = append(users, u)
}
if err := rows.Err(); err != nil {
return nil, err
}
return users, nil
}
Test
func TestGetUsersOk(t *testing.T) {
db, mock := NewMock()
mock.ExpectQuery("SELECT firstname, lastname, email, age FROM Users;").
WillReturnRows(sqlmock.NewRows([]string{"firstname", "lastname", "email", "age"}).
AddRow("pepe", "guerra", "pepe#gmail.com", 34))
subject := UserProvider{
DatabaseProvider: repositories.NewMockDBProvider(db, nil),
}
resp, err := subject.GetUsers()
assert.Nil(t, err)
assert.NotNil(t, resp)
assert.Equal(t, 1, len(resp))
}
func NewMock() (*sql.DB, sqlmock.Sqlmock) {
db, mock, err := sqlmock.New()
if err != nil {
log.Fatalf("an error '%s' was not expected when opening a stub database connection", err)
}
return db, mock
}
I find that writing tests against a live database makes for more high quality tests. The challenge with Postgres is that there's no good in-memory fake that you can substitute in.
What I came up with is standing up the postgres Docker container and creating temporary databases in there. The PostgresContainer type in the github.com/bitcomplete/sqltestutil package does exactly this:
# Postgres version is "12"
pg, _ := sqltestutil.StartPostgresContainer(context.Background(), "12")
defer pg.Shutdown(ctx)
db, err := sql.Open("postgres", pg.ConnectionString())
// ... execute SQL
Per the docs, it's a good idea to set up your tests so that the container is only started once, as it can take a few seconds to start up (more if the image needs to be downloaded). It suggests some approaches for mitigating that problem.
Context
Currently I have a REST API that manages customer's data in a db. I'm using the following stack:
Go 1.13
github.com/jinzhu/gorm v1.9.1
Postgres 11
I have the following connection settings.
// NewConnection ...
func NewConnection() (*gorm.DB, error) {
config := getConfig()
connStr := "host=xx.xx.xx port=5432 user=chavista-hatter dbname=my-db password=abc sslmode=verify-ca sslrootcert=/path/to/rcert sslcert=/path/to/cert sslkey=/path/to/key connect_timeout=0"
db, err := gorm.Open("postgres", conn)
if err != nil {
return nil, err
}
db.DB().SetMaxOpenConns(25)
db.DB().SetMaxIdleConns(25)
db.DB().SetConnMaxLifetime(5 * time.Minute)
db.SingularTable(true)
if config.LogQueries {
db = db.Debug()
}
return db, nil
}
I get a connection in the main class and inject that connection into a repository class that executes the queries through Gorm (ORM)
Main class
db, err := database.NewConnection()
if err != nil {
panic(fmt.Sprintf("failed to connect database --> %v", err))
}
fmt.Println("database connection established successfully")
defer db.Close()
customerRepo := customer.NewRepository(db)
Repository class
type repository struct {
db *gorm.DB
}
//NewRepository
func NewRepository(db *gorm.DB) Repository {
return &repository{
db: db,
}
}
func (r *repository) Register(customer *models.Customer) (string, error) {
err := r.db.Create(&customer).Error
if err != nil {
return "", err
}
return customer.key, nil
}
Problem
I sending over 500k request (INSERTS) to my db which have 512 connections available and after a few minutes the following error starts to come up repeatedly in postgres log:
unexpected EOF on client connection with an open transaction
could not receive data from client: Connection reset by peer
Any help?
How are you using GetConnection in your code? It's creating a new connection pool every time it's called - ideally you'd only want to call it once, and pass that single connection around wherever it's used.
I would try changing it to this:
var db *gorm.DB
func NewConnection() (*gorm.DB, error) {
if db != nil {
return db, nil
}
config := getConfig()
connStr := "host=xx.xx.xx port=5432 user=chavista-hatter dbname=my-db password=abc sslmode=verify-ca sslrootcert=/path/to/rcert sslcert=/path/to/cert sslkey=/path/to/key connect_timeout=0"
var err error
db, err = gorm.Open("postgres", conn)
if err != nil {
return nil, err
}
db.DB().SetMaxOpenConns(25)
db.DB().SetMaxIdleConns(25)
db.DB().SetConnMaxLifetime(5 * time.Minute)
db.SingularTable(true)
if config.LogQueries {
db = db.Debug()
}
return db, nil
}
and see if it solves the issue.
I am attempting to export a full SQL dump of one of our Cloud SQL Postgres instances to Google Cloud Storage so we can have more frequent backups using the google-api-go-client (https://github.com/googleapis/google-api-go-client)
Not matter what I configure, I keep getting this error: panic: googleapi: Error 403: The client is not authorized to make this request., notAuthorized from sqladminService.Instances.Export.
I have a service account configured with the following permissions:
Cloud SQL Admin
Storage Admin
Compute Storage Admin (for something else)
The bucket I am exporting to inherits permissions from the Storage Admin role.
Code:
./k8s/sql.go
package gcp
import (
"fmt"
"time"
"golang.org/x/net/context"
"golang.org/x/oauth2/google"
sqladmin "google.golang.org/api/sqladmin/v1beta4"
)
type SQLService interface {
Test(project string) error
}
type sqlService struct {
context context.Context
sqladminService *sqladmin.Service
}
func NewSQLService(serviceAccountJSON []byte) (SQLService, error) {
context := context.Background()
jwtCfg, err := google.JWTConfigFromJSON(serviceAccountJSON, sqladmin.SqlserviceAdminScope, sqladmin.CloudPlatformScope)
if err != nil {
return sqlService{}, err
}
httpClient := jwtCfg.Client(context)
sqladminService, err := sqladmin.New(httpClient)
if err != nil {
return sqlService{}, err
}
return sqlService{
context: context,
sqladminService: sqladminService,
}, nil
}
func (s sqlService) Test(project string) error {
instance := "REGION:INSTANCE_NAME
storageURI := fmt.Sprintf("gs://BUCKET/FILE-%s.sql.gz", time.Now().Format(time.RFC3339))
databases := []string{"DATABASE"}
req := &sqladmin.InstancesExportRequest{
ExportContext: &sqladmin.ExportContext{
Uri: storageURI,
Databases: databases,
},
}
_resp, err := s.sqladminService.Instances.Export(project, instance, req).Context(s.context).Do()
if err != nil {
return err
}
return nil
}
Test code:
func Test(cfg config.Config) {
sql, err := gcp.NewSQLService(cfg.GCPServiceAccountEncodedCreds)
if err != nil {
panic(err)
}
err = sql.Test(cfg.Project)
if err != nil {
panic(err)
}
}
Any help would be appreciated
The documentation for InstancesExport shows that the required parameters are the "projectId" and the "instanceId". You have declared instance as "REGION:INSTANCE_NAME" - but what you really want is "INSTANCE_NAME".
You aren't authorized to view that instance (because in this case, it doesn't exist).