TDD with database and Go - postgresql

I'm trying to wrap my head around test driven development with Go and having an issue testing my CRUD functions since they are written for my production database. I'm coming from Ruby on Rails so I am used to using a test database, but Go doesn't seem to be too friendly in this regard.
So, how does one go about testing CRUD with Go?
main.go
package main
import (
"database/sql"
)
type book struct {
id int `json:"id"`
isbn string `json:"isbn"`
title string `json:"title"`
author string `json:"author"`
price float32 `json:"price"`
}
// type Books []*Book
// CRUD functions for Book
func (b *book) getBook(db *sql.DB) error {
return db.QueryRow("SELECT * FROM books WHERE id=$1", b.id).Scan(&b)
}
app.go
func (a *App) Initialize(dbname string) {
var err error
a.DB, err = sql.Open("postgres", "postgresql://localhost:5432/bookstore?sslmode=disable")
if err != nil {
log.Fatal(err)
}
}
my test
func TestGetBook(t *testing.T) {
clearTable()
addBook(1)
req, _ := http.NewRequest("GET", "/book/1", nil)
response := executeRequest(req)
checkResponseCode(t, http.StatusOK, response.Code)
}
The problem is that this keeps on looking at the books table in my DB, not the books_test table I'd like to use for testing. How can I go about making ONLY the tests use the books_test DB?

You should create a dev/test database which should be a complete copy of your production database. You will never want to run test directly against your production database since too many unexpected issues could happen.
A workaround would be starting up your app first, which creates the connection to your database, then run the test. You can use IntelliJ to achieve this.
TDD in my opinion is great for developing business logic layer code since new models and business processes can have unexpected impacts on existing ones.

#Godzilla74, there'are 2 solutions: enable SSL for test DB (try to check database settings or ask your system administrator) of have completely different setting for test:
func (a *App) Initialize(dbname string) {
var err error
pgsettings := os.Getenv("PGSETTINGS")
if pgsettins == "" {
// default options if not overridden
pgsettins := "postgresql://localhost:5432/bookstore?sslmode=disable"
}
a.DB, err = sql.Open("postgres", pgsettins)
if err != nil {
log.Fatal(err)
}
}
So you can run set environment setting to any required value and run app, like so:
export PGSETTINGS="postgresql://localhost:5432/bookstore_test?sslmode=disable"
go run main.go

Related

Concurrent index creation fails when done in a Go program

I am trying to create some concurrent indexes using the command CRETAE INDEX CONCURRENTLY ..... through migrations in my golang project. But whenever I run that particular migration it just takes infinitely long and is never executed.
I went and checked for the logs of the POSTGRES DB and found this thing:
The weird thing is only in migrations I am not able to create concurrent indexes whereas in my main.go if i just directly write code to execute the query it is executing successfully and even on golang's DB query console it is able to create a index concurrently.
Here is my migration package code:
func NewGorm(d *gorm.DB) *GORM {
return &GORM{db: d}
}
func (g *GORM) Run(m Migrator, app, name, methods string, logger log.Logger) error {
g.txn = g.db.Begin()
ds := &datastore.DataStore{ORM: g.db}
if methods == UP {
err = m.Up(ds, logger)
} else {
err = m.Down(ds, logger)
}
if err != nil {
g.rollBack()
return &errors.Response{Reason: "error encountered in running the migration", Detail: err}
}
g.commit()
return nil
}
I know it has something to do with transactions, but i also tried disabling it by passing flag SkipDefaultTransaction: true when initializing the connection with GORM but that also didn't worked and results were the same.
Please help how can i create concurrent indexes in migrations using GORM.
Let me try to help you with the issue. First, you should update the question with all of the source code so we can understand better what's going on and help you in a more accurate way. Anyway, I'll try to help you with what we have so far. I was able to achieve your goal with the following code:
package main
import (
"fmt"
"gorm.io/driver/postgres"
"gorm.io/gorm"
)
type Post struct {
Id int
Title string `gorm:"index:idx_concurrent,option:CONCURRENTLY"`
}
type GORM struct {
db *gorm.DB
}
func main() {
dsn := "host=localhost user=postgres password=postgres dbname=postgres port=5432 sslmode=disable"
db, err := gorm.Open(postgres.Open(dsn), &gorm.Config{})
if err != nil {
panic(err)
}
db.AutoMigrate(&Post{})
m := db.Migrator()
if idxFound := m.HasIndex(&Post{}, "idx_concurrent"); !idxFound {
fmt.Println("idx missing")
return
}
fmt.Println("idx already present")
}
To achieve what you need, it should be enough to add a gorm annotation next to the field where you want to add the index (e.g. Title). In this annotation you can specify to create this index in a concurrent to avoid locking the table. Then, I used the gorm.Migrator to check for the index existence.
If you've already the table, you can simply add the annotation to the model struct definition and gorm will take care of it when you'll run the AutoMigrate method.
Thanks to this you should be able to cover all of the possible scenario you might face.
Let me know if this solves your issue or if you need something else, thanks!

I need advice on Go integration testing using Postgres db in Docker

I am very new to Go world. I have some db functions that I need to test.
So first I have a database.go file that connects to a postgres db:
import (
"fmt"
"gorm.io/driver/postgres"
"gorm.io/gorm"
"os"
)
var DB *gorm.DB
var err error
func Open() error {
dsn := fmt.Sprintf("host=%s user=%s password=%s dbname=%s port=%s sslmode=disable", os.Getenv("HOST"), os.Getenv("USER"),
os.Getenv("PASSWORD"), os.Getenv("DB"), os.Getenv("PORT"))
DB, err = gorm.Open(postgres.Open(dsn), &gorm.Config{})
if err != nil {
return err
}
return nil
}
Then I have a customers.go file with functions that interact with that db:
import (
"customers/cmd/database"
"time"
)
type Customers struct {
ID int
CustomerName string
Active bool
Balance float32
ActiveSince time.Time
}
func Get(id int) (Customers, error) {
var customer Customers
result := database.DB.First(&customer, id)
if result.Error != nil {
return Customers{}, result.Error
} else {
return customer, nil
}
}
This is all running in docker, there is customers container and postgres container. Now the question is how do I test my Get(id int) function? I was researching dockertest but that spins up a different db and my Get function uses the one I specified in database.go. So is there a standard Go way to test these functions?
it is a docker net problem but golang:
you can create a docker net, run both container in one net.doc
or use network --network=host
export postgres container's port to localhost, and customers container link to localhost,-pxx:xx
there is standard way of unit tesing in go. pl refer testing and testify/assert. Unit tests are typically written in xxx_test.go file next to the code.
coming to unit testing of db access layer code, one option would be to have a testenv helper and use it on these lines.
customers_test.go:
package dbaccess
import "testing"
func TestDbAccessLayer(t *testing.T) {
testEnv := testUtils.NewTestEnv()
// testEnv should do the required initialization e.g.
// start any mocked services, connection to database, global logger etc.
if err := testEnv.Start(); err != nil {
t.Fatal(err)
}
// at the end of the test, stop need to reset state
// e.g. clear any entries created by test in db
defer testEnv.Stop()
// add test code
// add required assertion to validate
}
have a separate docker-compose.yml file and use it with docker compose command to start/stop services like postgresdb.
go test command may be used to run the tests. refer to the command docs for details.

Run cron in Golang while having different databases

I am working on a SaaS based project on which Merchants can subscribe to set up their online store.
Project Overview
I am using Golang (backend), Mongodb database service and Angular4 (frontend) to build the system. I have multiple merchants that can set up their store. Each merchant has its own url (having its business name as subdomain in the url) to connect to his database.
For Routing, I am using Golang's Gin framework at back end.
Problem
I want to run the cron jobs for the merchant-specific database. In these cron jobs there are some operations that need to connect to the database. But in my routing, until a route for an API is called, the database won't be set. And ultimately, the cron does not run with proper data.
Code
cron.go
package cron
import (
"gopkg.in/robfig/cron.v2"
"controllers"
)
func RunCron(){
c := cron.New()
c.AddFunc("#every 0h1m0s", controllers.ExpireProviderInvitation)
c.Start()
}
Controller function
func ExpireProviderInvitation() {
bookingAcceptTimeSetting, _ := models.GetMerchantSetting(bson.M{"section": "providers", "option_name": "bookings_accept_time"})
if bookingAcceptTimeSetting.OptionValue != nil{
allInvitations, _ := models.GetAllBookingInvitations(bson.M{ "status": 0, "send_type": "invitation", "datetime": bson.M{"$le": float64(time.Now().Unix()) - bookingAcceptTimeSetting.OptionValue.(float64)} })
if len(allInvitations) > 0 {
for _, invitationData := range allInvitations {
_ = GetNextAvailableProvider(invitationData.Bid, invitationData.Pid)
}
}
}
}
router.go
func NewRouter() {
router := gin.Default()
router.Use(gin.Recovery())
router.Use(SetMerchantDatabase)
public := router.Group("/api/v1")
for _, route := range publicRoutes{
switch route.Method {
case "GET" : public.GET(route.Pattern, route.HandlerFunc)
case "POST" : public.POST(route.Pattern, route.HandlerFunc)
case "PUT" : public.PUT(route.Pattern, route.HandlerFunc)
case "DELETE": public.DELETE(route.Pattern, route.HandlerFunc)
default : public.GET(route.Pattern, func(c *gin.Context){
c.JSON(200, gin.H{
"result": "Specify a valid http method with this route.",
})
})
}
}
router.NoRoute(controllers.UnauthorizedAccessResponse)
router.Run(":8080")
}
func SetMerchantDatabase(c *gin.Context){
subdomain := strings.Split(c.Request.Host, ".")
if len(subdomain) > 0{
config.Database = subdomain[0]
config.CurrentBusinessName = subdomain[0]
}else{
errMsg := "Failed: Invalid domain in headers."
response := controllers.ResponseController{
config.FailureCode,
config.FailureFlag,
errMsg,
nil,
}
controllers.GetResponse(c, response)
c.Abort()
}
c.Next()
}
main.go
package main
import (
"cron"
)
func main(){
cron.RunCron()
NewRouter()
}
Explanation of above code
An example route can be:
Route{ "AddCustomer", "POST", "/customer", controllers.SaveCustomer },
An example API url can be:
http://business-name.main-domain.com/api/v1/customer
Where "business-name" is the database which is set whenever an API is called.
I want to run my cron without calling an API route.
Alternative approach
In Shell script, we can run cron by hitting url as a command. For this, I can create a url to run it as a command. But this is my theoratical approach. Also I don't know how will I get different merchant databases.
I am not sure if this approach will work. Any kind of help will be greatly appreciated.
You need to adapt SetMerchantDatabase to work independently of your router. Then you can have it set things for Cron just as well.

Go: Create io.Writer inteface for logging to mongodb database

Using go (golang):
Is there a way to create a logger that outputs to a database?
Or more precisely, can I implement some kind of io.Writer interface that I can pass as the first argument to log.New()?
EG: (dbLogger would receive the output of the log and write it to the database)
logger := log.New(dbLogger, "dbLog: ", log.Lshortfile)
logger.Print("This message will be stored in the database")
I would assume that I should just create my own database logging function, but I was curious to see if there is already a way of doing this using the existing tools in the language.
For some context, I'm using mgo.v2 to handle my mongodb database, but I don't see any io.Writer interfaces there other than in GridFS, which I think solves a different problem.
I'm also still getting my head around the language, so I may have used some terms above incorrecly. Any corrections are very welcome.
This is easily doable, because the log.Logger type guarantees that each log message is delivered to the destination io.Writer with a single Writer.Write() call:
Each logging operation makes a single call to the Writer's Write method. A Logger can be used simultaneously from multiple goroutines; it guarantees to serialize access to the Writer.
So basically you just need to create a type which implements io.Writer, and whose Write() method creates a new document with the contents of the byte slice, and saves it in the MongoDB.
Here's a simple implementation which does that:
type MongoWriter struct {
sess *mgo.Session
}
func (mw *MongoWriter) Write(p []byte) (n int, err error) {
c := mw.sess.DB("").C("log")
err = c.Insert(bson.M{
"created": time.Now(),
"msg": string(p),
})
if err != nil {
return
}
return len(p), nil
}
Using it:
sess := ... // Get a MongoDB session
mw := &MongoWriter{sess}
log.SetOutput(mw)
// Now the default Logger of the log package uses our MongoWriter.
// Generate a log message that will be inserted into MongoDB:
log.Println("I'm the first log message.")
log.Println("I'm multi-line,\nbut will still be in a single log message.")
Obviously if you're using another log.Logger instance, set the MongoWriter to that, e.g.:
mylogger := log.New(mw, "", 0)
mylogger.Println("Custom logger")
Note that the log messages end with newline as log.Logger appends it even if the log message itself does not end with newline. If you don't want to log the ending newline, you may simply cut it, e.g.:
func (mw *MongoWriter) Write(p []byte) (n int, err error) {
origLen := len(p)
if len(p) > 0 && p[len(p)-1] == '\n' {
p = p[:len(p)-1] // Cut terminating newline
}
c := mw.sess.DB("").C("log")
// ... the rest is the same
return origLen, nil // Must return original length (we resliced p)
}

Execute SparkCore function using Gobot.io and sleepy RESTful Framework for Go

I have the following bit of code where I'm using the RESTful framework for Go called sleepy.
I can successfully start the service at: http://localhost:3000, however when I try to access http://localhost:3000/temperature I'm expecting my SparkCore function dht to execute.
I'm using the Gobot.io Spark platform to execute this function based on this example, which I've implemented in my own code.
The problem is that the code doesn't get past the gobot.Start() method inside the Get() function so I can't actually return the result data.
I'm setting the data value hoping that I can do the:
return 200, data, http.Header{"Content-type": {"application/json"}}
But it never gets called becuase of the gobot.Start().
I'm very new to Go so any help would be greatly appreciated.
package main
import (
"net/url"
"net/http"
"fmt"
"github.com/dougblack/sleepy"
"github.com/hybridgroup/gobot"
"github.com/hybridgroup/gobot/platforms/spark"
)
var gbot = gobot.NewGobot()
var sparkCore = spark.NewSparkCoreAdaptor("spark", "device_id", "auth_token")
type Temperature struct {}
func (temperature Temperature) Get(values url.Values, headers http.Header) (int, interface{}, http.Header) {
work := func() {
if result, err := sparkCore.Function("dht", ""); err != nil {
fmt.Println(err)
} else {
data := map[string]string{"Temperature": result}
fmt.Println("result from \"dht\":", result)
}
}
robot := gobot.NewRobot("spark",
[]gobot.Connection{sparkCore},
work,
)
gbot.AddRobot(robot)
gbot.Start()
return 200, data, http.Header{"Content-type": {"application/json"}}
}
func main() {
api := sleepy.NewAPI()
temperatureResource := new(Temperature)
api.AddResource(temperatureResource, "/temperature")
fmt.Println("Listening on http://localhost:3000/")
api.Start(3000)
}
gbot.Start() is a blocking call.
In this context, you are expected to call it as:
go gbot.Start()
This will launch it in a goroutine (think thread) and then let your app continue.
When you look at the gobot example app, they don't run in the background since it is the main function. If main runs everything in the background and doesn't wait for anything, the app exits immediately with no apparent effect.