Re-usable components and fixtures in Golang tests - postgresql

I'm just getting started with Golang and writing my first test suite.
I have a background in Rails, which has fantastic support for testing tools (Rspec, Cucumber, etc..), so I'm approaching my golang tests with a similar mindset (not sure if that's the right or wrong thing to do)
I have a User data model (basically a struct) that reads records from a users table in postgres and stores an array of them. (Essentially a really simple version of what ActiveRecord does in the Rails world)
I'd like to write a test that checks if the routine correctly reads from the DB and builds the models.
In almost every test suite I'll be connecting to the DB so I have a helper named establish_db_connection. Where can I place this so that it's centrally available to all my tests?
Building off #1 - is there an equivalent of a before block or some setup/teardown method where I can establish a connection before every test?
Lastly, how do I handle fixtures? Right now before each test I call a clear_db function that resets all tables and inserts some static data rows. I'd love to move away from fixtures and use factories to build data as needed (very similar to FactoryGirl in Rails), but not sure how common that is in Golang.
Is the built-in go test framework the best approach, or are there better alternatives?

Go is based on strong package management, meaning a namespace is treated as one single file. If establish_db_connection is used within a single test package, it can begin with a lowercase letter to signify a private instance and use it in the test file with the same package as the code being tested (Note that naming convention in Go is establishDBConnection).
However, most of the time, as in data/sql, you will want to obtain a DB connection once and keep that around until the test is finished (more like a factory and injection pattern).
There is none in the standard testing package. If you like BDD, Goconvey use scopes to define fixtures and a reset function for teardown.
You can use factory and dependency injections in your testing. I think that's pretty idiomatic.
A few includes Goconvey, Ginkgo and Testify They all have pros and cons of their own. The first two often end up with too many nested scopes, but Goconvey has a great browser-based real-time testing server which can be used with Go standard testing.
Since there's no global variables/functions in Go, you might design your project in interface-delegate pattern to help with importing functions cross packages and avoiding cyclic imports when dealing with cross-package testing.
mypackage
type DBOptions struct {
Name, Credentials string
}
func aFunc(db *sql.DB) error {
// do something
return nil
}
func bFunc(db *sql.DB) int, error {
// do something
return 0, nil
}
func establishConn(opts *DBOptions) (*sql.DB, error) {
db, err := sql.Open(opts.Name, opts.Credentials)
if err != nil {
return nil, err
}
return db, nil
}
func destroyConn(conn *sql.DB) {
conn.Close()
}
// test file
mypackage
import "testing"
var myOpt = &DBOptions{
Name: "mysql",
Credentials: "user:password#tcp(127.0.0.1:3306)/hello",
}
var conn, _ = establishConn(myOpt)
func TestAFunc(t *testing.T) {
err := aFunc(conn)
if err != nil {
t.Error(err)
}
}
func TestBFunc(t *testing.T) {
err := aFunc(conn)
if err != nil {
t.Error(err)
}
}
// use `conn` in other tests ...
destroyConn(conn)

About test fixture library similar to FactoryGirl in Rails, There are some choices in go.
those two libraries got stars most.
https://github.com/bluele/factory-go
https://github.com/go-testfixtures/testfixtures
And I also implemented test-fixture library which is type-safe, DRY, and flexible compared to the above libraries!
https://github.com/k-yomo/fixtory

on fixtures: consider passing functions in your testcases:
package main
import "testing"
type testcase struct {
scenario string
before func(string)
after func()
input string
expOutput string
}
var state = ""
func setup(s string) {
state = s
}
func nilSetup(s string) {}
func reset() {
state = ""
}
func execute(s string) string {
return state
}
func TestSetupTeardown(t *testing.T) {
tcs := []testcase{
{
scenario: "blank output when initial state is wrong",
before: nilSetup,
after: reset,
input: "foo",
expOutput: "",
},
{
scenario: "correct output when initial state is right",
before: setup,
after: reset,
input: "foo",
expOutput: "foo",
},
}
for _, tc := range tcs {
tc.before(tc.input)
if out := execute(tc.input); out != tc.expOutput {
t.Fatal(tc.scenario)
}
tc.after()
}
}

I built a tiny utility library to make it easy to create reusable fixtures to tests in go. Check out https://github.com/houqp/gtest to see if it solves your problem.
Here is a quick example on how to create a database transaction fixture for each test in a test group:
type TransactionFixture struct{}
// Construct can take other fixtures as input parameter as well
func (s TransactionFixture) Construct(t *testing.T, fixtures struct{}) (*sqlx.Tx, *sqlx.Tx) {
tx := // create db transaction here
return tx, tx
}
func (s TransactionFixture) Destruct(t *testing.T, tx *sqlx.Tx) {
tx.Rollback()
}
func init() {
// register and make fixture available to all tests
gtest.MustRegisterFixture(
"Transaction", &TransactionFixture{}, gtest.ScopeSubTest)
}
// begin of test definition
type SampleTests struct{}
func (s *SampleTests) Setup(t *testing.T) {
// you can create/initialize DB in this method
// DB instance can also be implemented as a fixture and get injected into Transanction fixture.
}
func (s *SampleTests) Teardown(t *testing.T) {
// you can clean up all DB resources in this method
}
func (s *SampleTests) BeforeEach(t *testing.T) {}
func (s *SampleTests) AfterEach(t *testing.T) {}
func (s *SampleTests) SubTestFoo(t *testing.T, fixtures struct {
Tx sqlx.Tx `fixture:"Transaction"`
}) {
// transaction is available as fixtures.Tx in this test
}
func TestSampleTests(t *testing.T) {
gtest.RunSubTests(t, &SampleTests{})
}
See https://godoc.org/github.com/houqp/gtest and https://github.com/houqp/gtest/blob/master/example_test.go for more advanced and uptodate examples.

Related

Decode in BSON, an interface which is implemented by a private structure

I am trying to do a basic TODO list application in Go. I am creating the CRUD operations on my cluster from mongodb atlas. But I have a problem decoding BSON objects. For my model I use a struct which is unimported but it implements a interface which is used in the repo. When trying to read from database I get this error:
panic: no decoder found for interfaces.IToDoItem
I know I should somehow implement a decoder for my interface but can not realize how to do it, without exporting my main struct from model. That would also mean I won't have a privacy in the model and the items in the model can be accessed in any mode all around the program, a thing which I think is wrong.
Here is my code:
model.go
type toDoItem struct{
ItemId int `bson:"itemId,omitempty"`
Title string `bson:"title,omitempty"`
Description string `bson:"description,omitempty"`
}
func New(itemId int,title string,description string) interfaces.IToDoItem {
return toDoItem{
ItemId: itemId,
Title: title,
Description: description,
}
}
func (item toDoItem)GetItemId()int{
return item.ItemId
}
func (item toDoItem)GetTitle()string{
return item.Title
}
func (item toDoItem)GetDescription()string{
return item.Description
}
Interface
type IToDoItem interface {
GetItemId() int
GetTitle() string
GetDescription() string
}
repo function
func (r *Repository)GetAll() []interfaces.IToDoItem{
cursor, err := r.collection.Find(context.TODO(), bson.D{})
if err != nil{
panic(err)
}
defer cursor.Close(context.Background())
var allItems []interfaces.IToDoItem
for cursor.Next(context.Background()){
var result interfaces.IToDoItem
err := cursor.Decode(&result)
if err!= nil{
panic(err)
}
allItems = append(allItems[:],result)
}
fmt.Println(allItems)
return []interfaces.IToDoItem{}
}
For now it does not return anything because I want to resolve the issues at the repo level.
Interfaces are not concrete types, there may be multiple (any number of) types implementing it. The driver rightfully doesn't know how to instantiate it.
Fields of toDoItem are exported, there is no reason to make toDoItem itself unexported. Just export it and use a slice of []ToDoItem (or []*ToDoItem), and all your problems go away.
Also note that there is a Cursor.All() method to get all results, so you don't have to iterate over the documents and call Cursor.Decode() for each.
If you'd ever need to register custom decoders, check out this answer how to do it:
How to ignore nulls while unmarshalling a MongoDB document?

Validate field based on value of a different field with validator.v2

I have the following (simplified) struct:
type newAppRegister struct {
SomeFlag *bool `json:"someflag" validate:"nonnil"`
ComputeLevel string `json:"compute-level" validate:"computelevelvalidator"`
}
And computelevelvalidator is some validation function.
I want that if SomeFlag is false, then ComputeLevel will be required and run his validation function.
A possible solution is to create a static variable, and set it in a custom validation function of SomeFlag, so for example:
var someFlag bool
func someFlagValidator(v interface{}, param string) error {
st := reflect.ValueOf(v)
if st.Kind() != reflect.Bool {
return fmt.Errorf("must be bool")
}
someFlag = st.Bool()
return nil
}
And then in computelevelvalidator it's possible to condition based on his value.
But since this is a REST API, and it may receive plenty of calls per second, I'm not sure if using this static variable will work (I'm worried about race conditions - but I'm not quite sure about it).
I'm using go1.11 and validation package gopkg.in/validator.v2 (version gopkg.in/validator.v2 v2.0.0-20190827175613-1a84e0480e5b).
So first up: Yes, using this static (global) variable will be a problem for concurrent use/access. It's not the way to go.
If your validation logic is contextual (like you said: if the flag is false, then certain restrictions apply on another field), then implementing a simple Validate function is pretty straightforward. I know the package you use supports this type of thing, but an external package will almost always be designed to be fairly generic. At the very least you'll end up performing type assertions. Last I checked, you still need an explicit call for the validation validator.Validate(), so why not move that to a method on your type?
type newAppRegister struct {} // your type
func (n newAppRegister) Validate() error {
if err := validator.Validate(n); err != nil {
return err
}
// at this point, we now the flag field isn't nit, because it passed validation
if !*n.SomeFlag {
// validate ComputeLevel here
}
return nil
}
No need for type assertions, let alone reflection. With just these two fields, I'd even argue there's no need for the validator package at all. You can get the same thing done with just standard JSON tags:
type Foo struct {
SomeFlag *bool `json:"some_flag,omitempty"`
ComputeLevel string `json:"compute_level"`
}
func (f Foo) Validate() error {
if f.SomeFlag == nil {
return ErrSomeFlagRequired
}
if !*f.SomeFlag {
// validate ComputeLevel
}
return nil
}
It's fairly straightforward to use:
f := Foo{}
if err := json.Unmarshal([]byte(payload), &f); err != nil {
// some shady JSON was submitted
}
if err := f.Validate(); err != nil {
// JSON was technically valid, but payload made no sense
}
// handle valid request

How to put mongodb sessions in creachadair/jrpc2 Handlers

I want to build a microservice that uses jrpc2 and mongodb to manage small IoT devices. These are the libraries in question:
https://godoc.org/bitbucket.org/creachadair/jrpc2
https://godoc.org/github.com/globalsign/mgo
The problem is, being rather new to Golang in general, I'm not sure how to combine these things together. Here's the code I have:
func DeviceAdd(ctx context.Context) (map[string]string, error) {
m := make(map[string]string)
m["token"] = "0xdeadbeef"
return m, nil
}
func DeviceBootstrap(ctx context.Context, params map[string]string) (map[string]string, error) {
m := make(map[string]string)
m["entered_token"] = params["token"]
return m, nil
}
...
func NewServer() *jrpc2.Server {
assigner := jrpc2.MapAssigner{
"device_add": jrpc2.NewHandler(DeviceAdd),
"device_bootstrap": jrpc2.NewHandler(DeviceBootstrap),
"device_update": jrpc2.NewHandler(DeviceUpdate),
"device_get_status": jrpc2.NewHandler(DeviceGetStatus),
}
srv := jrpc2.NewServer(assigner, nil)
return srv
}
Basically, you write some basic functions that accept a context.Context or any other JSON-RPC params, and jrpc2.NewHandler(function) will attach them to the jrpc2.Server.
The problem is, I cannot change the function handler signatures to put a mgo.Session in there. Or should I make DeviceAdd/DeviceBootstrap methods of a struct, which has access to a mgo.Session?
According to another StackOverflow question, it seems I should make DeviceAdd/DeviceBootstrap methods of a struct that has an mgo.Session, but I'm really not sure if I understood it correctly, or if jrpc2.NewHandler will accept the newly converted methods, or if it's the right thing to do. Also, Golang's context.Context seems like the right place to put such a thing. Help!
A request can be handled by any value that implements the jrpc2.Handler interface. Perhaps the simplest way to adapt an mgo.Session to this interface would be to wrap it in a struct, e.g.,
type SessionHandler struct {
mgo.Session
}
func (s SessionHandler) Handle(ctx context.Context, req *jrpc2.Request) (interface{}, error) {
// whatever you want your handler to do, using the s.Session
// as a delegate.
}
Then you can write something like:
a := handler.Map{"DoAThing": SessionHandler{s}}
to plug it into your server.

Go language - running the same code for multiple select cases

Assuming we have multiple notification sources (channels) and we want to perform some similar tasks on some of these notification events. For example, consider the case where we call doSomething() every 10 minutes and also upon user request. A possible implementation in Go could be something like this:
var ticker = time.NewTicker(10*time.Minute)
nowDoSomething := make(chan time.Time, 1)
for {
select {
case <-ticker.C:
nowDoSomething<-time.Now()
case <-userReq: // some channel activated occasionally
nowDoSomething<-time.Now()
case <-nowDoSomething:
doSomething()
}
}
What would be other, perhaps better, ways of achieving a similar functionality?
Perhaps you could do this using the Select function from the reflect package:
func Select(cases []SelectCase) (chosen int, recv Value, recvOK bool)
Why signaling to yourself using a channel if you can just call doSomething()? I understand do Something might not be exactly a function, but you can always use a local closure.
See a working example:
package main
import (
"fmt"
"time"
)
func someRoutine (userReq chan int) {
doSomething := func () {
fmt.Printf ("OK %v\n", time.Now())
}
var ticker = time.NewTicker(10*time.Second)
for {
select {
case <-ticker.C:
doSomething()
case <-userReq: // some channel activated occasionally
doSomething()
}
}
}
func main() {
userReq := make(chan int, 1)
go someRoutine(userReq);
userReq <- 1 // user requested some action here (just to mark the start)
time.Sleep(15*time.Second)
userReq <- 1 // user requested some action here
time.Sleep(30*time.Second)// Put your own waiting and exiting logic here
}
If you still want to optimize it beyond this simple correction, there are always some alternatives available. But with just two channels to read from, make sure you're not doing premature optimization.

Create a function which returns a mongo collection with Golang and MGO

I'm new to GOLANG - I would like to reduce the complexity of the handlers in my GO API. I'm coming from a Node.js background!
At the moment a route handler looks like this:
func getCards(c web.C, w http.ResponseWriter, r *http.Request) {
session := *MongoConnection().CreateSession()
defer session.Close()
collection := session.DB(db).C(cardsCollection)
result := []Card{}
err := collection.Find(bson.M{}).All(&result)
if err != nil {
panic(err)
}
w.Header().Set("Content-Type", "application/json")
encoder := json.NewEncoder(w)
encoder.Encode(result)
}
What I would like to do is return a collection for use without having to do this part:
session := *MongoConnection().CreateSession()
defer session.Close()
collection := session.DB(db).C(cardsCollection)
Instead I would like to do something like
collection := Card.Collection()
And have it create the session etc, Is this possible?
Why don't you have the session created in your main function, and pass it to the packages that need it. This is obviously missing a ton of stuff but the general idea would be
package main
//imports
func main() {
session := *MongoConnection().CreateSession()
defer session.Close()
Card.SetSession(session)
//other stuff
log.Fatal(http.ListenAndServe(":80", nil))
}
Then in Card
package Card
var session *mgo.Session
func SetSession(s *mgo.Session) {
session = s
}
func (c *Card) Collection() *mgo.Collection {
return session.DB(db).C(cardsCollection)
}
What you are describing is a factory pattern. But there are caveats to deal with.
Defer is local to scope it is called. So in case you would put the defer into a factory method, the session would be closed basically when it is returned.
Simply defining just one session und reuse it all over the place is pretty much the same as using a single SQL database connection and not a pool - it scales horribly.
Here is what I tend to do: I have my collections as global, and do the following at the beginning of my method
// Create a func-local session
myCollection := globalCollection.With( globalCollection.Database.Session.Copy() )
// Close it on exiting the scope
defer myCollection.Database.Session.Close()
Of course, you could externalize the first line into a factory, but that would not really declutter the code or make it more readable.