Unit Testing Postgres db connection golang - postgresql

I am expected to have 80% test coverage even for pushing the basic project structure. I am a bit confused how do I write unit tests for the following code to Connect to postgres db and ping postgres for health check. Can someone help me please.
var postgres *sql.DB
// ConnectToPostgres func to connect to postgres
func ConnectToPostgres(connStr string) (*sql.DB, error) {
db, err := sql.Open("postgres", connStr)
if err != nil {
log.Println("postgres-client ", err)
return nil, err
}
postgres = db
return db, nil
}
// PostgresHealthCheck to ping database and check for errors
func PostgresHealthCheck() error {
if err := postgres.Ping(); err != nil {
return err
}
return nil
}
type PostgresRepo struct {
db *sql.DB
}
// NewPostgresRepo constructor
func NewPostgresRepo(database *sql.DB) *PostgresRepo {
return &PostgresRepo{
db: database,
}
}

You need to use this : https://github.com/DATA-DOG/go-sqlmock
Its very easy to use. Here is an example where a controller is getting tested using a mocked SQL :
Implementation
func (up UserProvider) GetUsers() ([]models.User, error) {
var users = make([]models.User, 0, 10)
rows, err := up.DatabaseProvider.Query("SELECT firstname, lastname, email, age FROM Users;")
if err != nil {
return nil, err
}
for rows.Next() {
var u models.User = models.User{}
err := rows.Scan(&u.Name, &u.Lastname, &u.Email, &u.Age)
if err != nil {
return nil, err
}
users = append(users, u)
}
if err := rows.Err(); err != nil {
return nil, err
}
return users, nil
}
Test
func TestGetUsersOk(t *testing.T) {
db, mock := NewMock()
mock.ExpectQuery("SELECT firstname, lastname, email, age FROM Users;").
WillReturnRows(sqlmock.NewRows([]string{"firstname", "lastname", "email", "age"}).
AddRow("pepe", "guerra", "pepe#gmail.com", 34))
subject := UserProvider{
DatabaseProvider: repositories.NewMockDBProvider(db, nil),
}
resp, err := subject.GetUsers()
assert.Nil(t, err)
assert.NotNil(t, resp)
assert.Equal(t, 1, len(resp))
}
func NewMock() (*sql.DB, sqlmock.Sqlmock) {
db, mock, err := sqlmock.New()
if err != nil {
log.Fatalf("an error '%s' was not expected when opening a stub database connection", err)
}
return db, mock
}

I find that writing tests against a live database makes for more high quality tests. The challenge with Postgres is that there's no good in-memory fake that you can substitute in.
What I came up with is standing up the postgres Docker container and creating temporary databases in there. The PostgresContainer type in the github.com/bitcomplete/sqltestutil package does exactly this:
# Postgres version is "12"
pg, _ := sqltestutil.StartPostgresContainer(context.Background(), "12")
defer pg.Shutdown(ctx)
db, err := sql.Open("postgres", pg.ConnectionString())
// ... execute SQL
Per the docs, it's a good idea to set up your tests so that the container is only started once, as it can take a few seconds to start up (more if the image needs to be downloaded). It suggests some approaches for mitigating that problem.

Related

Go Echo: POST Method gives Error "Method not allowed"

Building an app with echo and basically created some routes.
The GET ones are working fine, but the post one is give me the error:
Do not really understand where the error lies here.
{...."method":"GET","uri":"/addPerson", message=Method Not Allowed","...."bytes_in":0,"bytes_out":33}
main.go snippet
func initEchoServer() {
e := echo.New()
e.Use(middleware.Logger())
e.Use(middleware.Recover())
// get all persons
e.GET("/persons", Info)
// get specific id
e.GET("/persons/:id", getPerson)
e.POST("/addPerson", addPerson)
e.Logger.Fatal(e.Start(viper.GetString("port")))
}
func addPerson(c echo.Context) error {
ctx := context.Background()
db, err := sql.Open("postgres", "host=postgres port=5432 user=postgres dbname=postgres password=postgres sslmode=disable")
if err != nil {
log.Fatal(err)
}
queries := postgres.New(db)
insertedPerson, err := queries.CreatePersons(ctx, postgres.CreatePersonsParams{
Firstname: "Mike",
Lastname: "Jordan",
})
if err != nil {
log.Errorf("Failed to insert a person %v", err)
return err
}
fmt.Println(insertedPerson)
return c.JSONPretty(http.StatusOK, insertedPerson, " ")
}
queries.sql.go snippet
type CreatePersonsParams struct {
Firstname string
Lastname string
}
func (q *Queries) CreatePersons(ctx context.Context, arg CreatePersonsParams) (Person, error) {
row := q.db.QueryRowContext(ctx, createPersons, arg.Firstname, arg.Lastname)
var i Person
err := row.Scan(&i.ID, &i.Firstname, &i.Lastname)
return i, err
}
you're use post method in routers
e.POST("/addPerson", addPerson)
You can use postman to hit API using POST method, don't use browser
If you register routes with POST in echo, it will only register POST method on that path. But it seems that you GET that path.
You can use e.GET().

Why I get an error "client disconnected" when trying to get documents from mongo collection in go?

I have mongo capped collection and a simple API, written on Go. I built and run it. When I try to sent Get request or simply go localhost:8000/logger in browser - my process closes. Debug shows this happens, while executing "find" in collection. It produces error "client is disconnected". Collection has 1 document, and debug shows it is connected with my helper.
Go version 1.13
My code:
func main() {
r := mux.NewRouter()
r.HandleFunc("/logger", getDocs).Methods("GET")
r.HandleFunc("/logger", createDoc).Methods("POST")
log.Fatal(http.ListenAndServe("localhost:8000", r))
}
func getDocs(w http.ResponseWriter, r *http.Request) {
w.Header().Set("Content-Type", "application/json")
var docs []models.Logger
//Connection mongoDB with helper class
collection := helper.ConnectDB()
cur, err := collection.Find(context.TODO(), bson.M{})
if err != nil {
helper.GetError(err, w)
return
}
defer cur.Close(context.TODO())
for cur.Next(context.TODO()) {
var doc models.Logger
err := cur.Decode(&doc)
if err != nil {
log.Fatal(err)
}
docs = append(docs, doc)
}
if err := cur.Err(); err != nil {
log.Fatal(err)
}
json.NewEncoder(w).Encode(docs)
}
func ConnectDB() *mongo.Collection {
client, err := mongo.NewClient(options.Client().ApplyURI("mongodb://127.0.0.1:27017"))
if err != nil {
log.Fatal(err)
}
fmt.Println("Connected to MongoDB!")
logCollection := client.Database("local").Collection("loggerCollection")
return logCollection
}
According to the documentation, the call to mongo.NewClient doesn't ensure that you can connect the Mongo server. You should first call mongo.Client.Ping() to verify if you can connect to the database or not.
client, err := mongo.NewClient(options.Client().ApplyURI("mongodb://127.0.0.1:27017"))
if err != nil {
log.Fatal(err)
}
if err := client.Ping(context.TODO(), readpref.Primary()); err != nil {
// Can't connect to Mongo server
log.Fatal(err)
}
There could be several reasons behind failing to connect, the most obvious one is incorrect setup of ports. Is your mongodb server up and listening on port 27017? Is there any change you're running mongodb with Docker and it's not forwarding to the correct port?
I faced similar issue , read #Jay answer it definitely helped , as I checked my MongoDB was running using "MongoDB Compass" , then I changed the location of my insert statement , previously I was calling before the call of "context.WithTimeout". Below is working code.
package main
import (
"context"
"log"
"time"
"go.mongodb.org/mongo-driver/mongo"
"go.mongodb.org/mongo-driver/mongo/options"
)
type Book struct {
Name string `json:"name,omitempty"`
PublisherID string `json:"publisherid,omitempty"`
Cost string `json:"cost,omitempty"`
StartTime string `json:"starttime,omitempty"`
EndTime string `json:"endtime,omitempty"`
}
func main() {
client, err := mongo.NewClient(options.Client().ApplyURI("mongodb://localhost:27017"))
if err != nil {
log.Fatal(err)
}
ctx, _ := context.WithTimeout(context.Background(), 10*time.Second)
err = client.Connect(ctx)
if err != nil {
log.Fatal(err)
}
defer client.Disconnect(ctx)
testCollection := client.Database("BooksCollection").Collection("BooksRead")
inserRes, err := testCollection.InsertOne(context.TODO(), Book{Name: "Harry Potter", PublisherID: "IBN123", Cost: "1232", StartTime: "2013-10-01T01:11:18.965Z", EndTime: "2013-10-01T01:11:18.965Z"})
log.Println("InsertResponse : ", inserRes)
log.Println("Error : ", err)
}
I can see document inserted in console as well as in "MongoDB Comapass."
In heiper function "ConnectDB" after "NewClient" I must use "client.Connect(context.TODO())"
before any other use of client

Gorm & Postgres Connection reset by peer

Context
Currently I have a REST API that manages customer's data in a db. I'm using the following stack:
Go 1.13
github.com/jinzhu/gorm v1.9.1
Postgres 11
I have the following connection settings.
// NewConnection ...
func NewConnection() (*gorm.DB, error) {
config := getConfig()
connStr := "host=xx.xx.xx port=5432 user=chavista-hatter dbname=my-db password=abc sslmode=verify-ca sslrootcert=/path/to/rcert sslcert=/path/to/cert sslkey=/path/to/key connect_timeout=0"
db, err := gorm.Open("postgres", conn)
if err != nil {
return nil, err
}
db.DB().SetMaxOpenConns(25)
db.DB().SetMaxIdleConns(25)
db.DB().SetConnMaxLifetime(5 * time.Minute)
db.SingularTable(true)
if config.LogQueries {
db = db.Debug()
}
return db, nil
}
I get a connection in the main class and inject that connection into a repository class that executes the queries through Gorm (ORM)
Main class
db, err := database.NewConnection()
if err != nil {
panic(fmt.Sprintf("failed to connect database --> %v", err))
}
fmt.Println("database connection established successfully")
defer db.Close()
customerRepo := customer.NewRepository(db)
Repository class
type repository struct {
db *gorm.DB
}
//NewRepository
func NewRepository(db *gorm.DB) Repository {
return &repository{
db: db,
}
}
func (r *repository) Register(customer *models.Customer) (string, error) {
err := r.db.Create(&customer).Error
if err != nil {
return "", err
}
return customer.key, nil
}
Problem
I sending over 500k request (INSERTS) to my db which have 512 connections available and after a few minutes the following error starts to come up repeatedly in postgres log:
unexpected EOF on client connection with an open transaction
could not receive data from client: Connection reset by peer
Any help?
How are you using GetConnection in your code? It's creating a new connection pool every time it's called - ideally you'd only want to call it once, and pass that single connection around wherever it's used.
I would try changing it to this:
var db *gorm.DB
func NewConnection() (*gorm.DB, error) {
if db != nil {
return db, nil
}
config := getConfig()
connStr := "host=xx.xx.xx port=5432 user=chavista-hatter dbname=my-db password=abc sslmode=verify-ca sslrootcert=/path/to/rcert sslcert=/path/to/cert sslkey=/path/to/key connect_timeout=0"
var err error
db, err = gorm.Open("postgres", conn)
if err != nil {
return nil, err
}
db.DB().SetMaxOpenConns(25)
db.DB().SetMaxIdleConns(25)
db.DB().SetConnMaxLifetime(5 * time.Minute)
db.SingularTable(true)
if config.LogQueries {
db = db.Debug()
}
return db, nil
}
and see if it solves the issue.

Not able to use mongodb transactions using mongo go driver

I have created a mongodb replica set. I am able to run transactions from the mongo shell. But when I try to do it using mongo-go-driver I always get this error (IllegalOperation) Transaction numbers are only allowed on a replica set member or mongos. I am not sure where I am going wrong. I am using this as a reference https://github.com/simagix/mongo-go-examples/blob/master/examples/transaction_test.go.
I create the client like this
client, err := mongo.NewClient(options.Client().ApplyURI("mongodb://localhost:27017,localhost:27018,localhost:27019?replicaSet=rs"))
I can connect to the individual mongodb instances, just not the replica set.
This is the transaction that I am trying to run
var session mongo.Session
coll := db.Collection("collectionname")
if session, err = client.StartSession(); err != nil {
return "", fmt.Errorf("Could not start session: %q", err)
}
if err = session.StartTransaction(); err != nil {
return "", fmt.Errorf("Could not start Transaction: %q", err)
}
if err = mongo.WithSession(ctx, session, func(sc md.SessionContext) error {
newVal, err = coll.InsertOne(sc, val) // some val that I have
if err != nil {
sc.AbortTransaction(sc)
return fmt.Errorf("Error during New address creation, aborting: %q", err)
}
if err = sc.CommitTransaction(sc); err != nil {
return fmt.Errorf("Error While commiting New address Transaction: %q", err)
}
return nil
}); err != nil {
return "", err
}
session.EndSession(ctx)
Is there something I am missing. Is there some other example maybe that I can reference. Thanks for the help/suggestions.
This is an issue with your connection code - not your transaction implementation most likely. Try using the more modern connection string for connecting to a replica set:
connectionString := "mongodb+srv://USERNAME:PASSWORD#mongoatlas.1mxpg.mongodb.net/?retryWrites=true&w=majority"
var err error
Client, err = mongo.NewClient(options.Client().ApplyURI(ConnectionString))
if err != nil {
log.Fatal(err)
}

What am I not getting about Go sql query with variables?

I'm brand new to Go, and I've started working on some postgres queries, and I'm having very little luck.
I have a package that's just going to have some database queries in it. Here's my code.
main.go
package main
import (
"fmt"
)
func main() {
fmt.Println("Querying data")
myqueries.SelectAll("mytable")
}
myqueries.go
package myqueries
import (
"database/sql"
"fmt"
)
func SelectAll (table string) {
db, err := sql.Open("postgres","user=postgres dbname=mydb sslmode=disable")
if err != nil {
fmt.Println(err)
}
defer db.Close()
rows, err := db.Query("SELECT * FROM $1", table)
if err != nil {
fmt.Println(err)
} else {
PrintRows(rows)
}
}
func PrintRows(rows *sql.Rows) {
for rows.Next() {
var firstname string
var lastname string
err := rows.Scan(&firstname, &lastname)
if err != nil {
fmt.Println(err)
}
fmt.Println("first name | last name")
fmt.Println("%v | %v\n", firstname, lastname)
}
}
The error I get is pq: syntax error at or near "$1"
which is from myqueries.go file in the db.Query.
I've tried several variations of this, but nothing has worked yet. Any help is appreciated.
It looks like you are using https://github.com/lib/pq based on the error message and it's docs say that
pq uses the Postgres-native ordinal markers, as shown above
I've never known a database engine that allows the parameterized values in anything other than values. I think you are going to have to resort to string concatenation. I don't have a Go compiler available to me right now, but try something like this. Because you are inserting the table name by concatination, you need it sanitized. pq.QuoteIdentifier should be able to help with that.
func SelectAll (table string) {
db, err := sql.Open("postgres","user=postgres dbname=mydb sslmode=disable")
if err != nil {
fmt.Println(err)
}
defer db.Close()
table = pq.QuoteIdentifier(table)
rows, err := db.Query(fmt.Sprintf("SELECT * FROM %v", table))
if err != nil {
fmt.Println(err)
} else {
PrintRows(rows)
}
}
EDIT: Thanks to hobbs to pointing out pq.QuoteIdentifier